[suntectec]

What is the purpose of this pull request?

Add a DB2 pipeline connector so the YAML Pipeline API can read from IBM DB2, alongside the existing MySQL / Postgres / Oracle / SQL Server pipeline connectors. Before this change DB2 was only reachable from the DataStream / SQL APIs through flink-connector-db2-cdc.

JIRA: FLINK-40508

Brief change log

• New module flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-db2, registered in the pipeline-connectors parent pom.xml.
• Db2DataSourceFactory (identifier db2) plus Db2DataSource, Db2PipelineSource, Db2EventDeserializer, Db2MetadataAccessor, Db2PipelineRecordEmitter, Db2SchemaDataTypeInference, Db2SchemaUtils and Db2TypeUtils.
• Startup and snapshot behaviour reuse the existing source connector: scan.startup.mode accepts initial (default) and latest-offset, captured tables are declared as schema.table.
• jdbc.properties.* are forwarded to Debezium as database.* connector properties, and an explicitly configured debezium.* property wins. A test reads VALUES CURRENT SCHEMA back from a real connection to prove the properties reach JDBC.
• Type mapping preserves length, precision and scale for CHAR/VARCHAR, BINARY/VARBINARY, integer, floating point, DECIMAL/NUMERIC, DATE, TIME and TIMESTAMP. Types Debezium cannot materialise reliably (DECFLOAT, XML, CLOB, DBCLOB, BLOB) fail fast during schema inference instead of being silently advertised, so schema and record fields cannot drift apart.
• The IBM JDBC driver is excluded from the shaded artifact because of its license; installing it into FLINK_HOME/lib of the submitting client and every JM/TM is documented.
• Capability boundaries are documented explicitly: schema-change.enabled only forwards schema change events reported by the underlying source and does not enable runtime DDL discovery, checkpoint/savepoint restore does not refresh an outdated schema, and UPDATE events require CHG_UPD_TO_DEL_INS='Y' on the capture table because direct U records are not supported.
• .github/workflows/modules.py adds the module to MODULES_DB2, so CI builds and tests it for Flink 1.20 and Flink 2.x; .github/labeler.yml adds the db2-pipeline-connector rule.
• Documentation in docs/content and docs/i18n/zh, plus the pipeline-connectors overview pages and the Docusaurus sidebar.

flink-connector-db2-cdc is not modified. src/test/resources/db2_server/ is copied from the source connector because Db2TestBase resolves the Docker build context through Paths.get(url.toURI()), which cannot read resources out of a test-jar.

Verifying this change

This change added tests and can be verified as follows:

• Db2TypeUtilsTest covers the supported mappings, length and precision handling, and the rejection path for unsupported types.
• Db2DataSourceFactoryITCase (13 cases) runs against a real DB2 instance through Testcontainers, including jdbc.properties.* / debezium.* precedence.
• Db2PipelineITCase (5 cases) covers initial with and without schema changes, an 18-column full-type table for snapshot plus streaming INSERT/UPDATE/DELETE, and fail-fast on the UNSUPPORTED_TYPES fixture.
• mvn verify passes on Java 11 / Flink 1.20.3 and Java 17 / Flink 2.2.0 (-Pflink2): 26 tests per generation, plus Spotless and Checkstyle.
• Documentation verified with npm test and npm run build for both locales.

Documentation

• Does this pull request introduce a new feature? yes
• If yes, how is the feature documented? docs

---

Was generative AI tooling used to co-author this PR?

• Yes (Qoder)

Generated-by: Qoder

[Ho1ySh1tC0d1ng]

Hi @suntectec, thanks for working on the DB2 pipeline connector! I’m the author of #4454, which explored the same feature. After comparing the implementations and tracing the shared DB2 source code, I noticed a few issues that may be worth addressing before merging:

1. DECFLOAT support appears incomplete.
   Db2TypeUtils exposes DECFLOAT columns in the pipeline schema, but the default Debezium 1.9.8 DB2 value converter does not handle DECFLOAT reported as JDBC Types.OTHER. Debezium therefore ignores these columns when building its record schema, creating a mismatch with the emitted CreateTableEvent. Also, mapping DECFLOAT(34) to DECIMAL(34, 0) cannot preserve fractional values. Could you check the complete schema/value conversion path and add tests with actual DECFLOAT data?

2. Runtime schema-change discovery is still disabled in the underlying source.
   Although schema-change.enabled defaults to true and the deserializer handles ALTER events, GET_LIST_OF_NEW_CDC_ENABLED_TABLES still contains AND 1=0. This prevents that discovery path from refreshing capture tables and scheduling schema migration after a schema change and recapture. It would be helpful to either complete and verify this path or clearly document the supported limitations.

3. jdbc.properties.* is accepted but does not appear to reach the connection configuration.
   The factory allows this prefix and the documentation advertises it, but configuration construction only extracts debezium.* properties. Could you add the JDBC-property mapping and verify the resulting connection parameters?

4. The shaded artifact includes the DB2 JDBC driver.
   The shade whitelist explicitly includes com.ibm.db2.jcc:db2jcc, while the existing DB2 connector documentation requires users to provide the driver separately. Could you check the packaging boundary and document the corresponding deployment steps?

5. The UPDATE capture prerequisites need to be explicit.
   The underlying query handles D/I records and paired updates, but not a direct IBMSNAP_OPERATION='U'. The test setup uses CHG_UPD_TO_DEL_INS=Y, so it does not exercise this case. Supporting direct U records, or documenting and validating the required capture setting, would help avoid failures in other DB2 setups.

Some of these are inherited limitations of the shared DB2 source rather than regressions introduced by this PR. However, they affect the capabilities exposed by the new pipeline connector.

Tests covering real DECFLOAT values, ALTER/recapture followed by DML, and checkpoint/restart across schema changes would be particularly valuable. Thanks again for the contribution—I hope these observations help strengthen the implementation!

[suntectec]

Re: the review above (@Ho1ySh1tC0d1ng)

Thanks for the detailed review. All five points are addressed in the reworked branch (rebased on master, single commit 7c54546):

1. DECFLOAT — instead of advertising a mapping Debezium 1.9.8 cannot materialise, Db2TypeUtils now fails fast during schema inference for DECFLOAT, XML, CLOB, DBCLOB and BLOB. Db2PipelineITCase#testUnsupportedColumnTypesFailFast reads real DECFLOAT(16) / DECFLOAT(34) values from DB2 and asserts the rejection, so the schema/record mismatch cannot occur silently.
2. Runtime schema-change discovery — not enabled. We documented the boundary instead: GET_LIST_OF_NEW_CDC_ENABLED_TABLES still contains AND 1=0, schema-change.enabled only forwards events reported by the underlying source, the initial CreateTableEvent is still emitted, and checkpoint/savepoint restore does not refresh an outdated schema.
3. jdbc.properties.* — now forwarded to Debezium as database.*, with an explicit debezium.* property taking precedence. Db2DataSourceFactoryITCase#testJdbcPropertiesAffectConnection reads VALUES CURRENT SCHEMA back from a real JDBC connection to prove the settings arrive.
4. Shaded driver — com.ibm.db2.jcc:* is excluded from the shaded artifact; the built jar contains no com/ibm/db2 classes and no IBM Maven metadata. The docs explain installing the driver into FLINK_HOME/lib of the submitting client and every JM/TM.
5. UPDATE prerequisites — documented: capture tables require CHG_UPD_TO_DEL_INS='Y', and direct IBMSNAP_OPERATION='U' records are not supported. The IT cases use the paired D/I setup you mentioned.

Test status: 26 cases per generation (8 unit + 13 factory IT + 5 pipeline IT) pass on Java 11 / Flink 1.20.3 and Java 17 / Flink 2.2.0, plus Spotless and Checkstyle; the documentation builds for both locales.

[suntectec]

@yuxiqian sorry for the direct ping. The three workflow runs (Flink CDC CI, Build documentation, Labeler-Trigger) are still in action_required, so no checks have started on the latest commit. Could you approve them, or point me to whoever is on review duty for pipeline connectors?

One more thing: the labeler rule added in this PR expects a db2-pipeline-connector label that does not exist in the repository yet. Creating it would let the rule apply.