Skip to content
maxnilzPublic

About

duckdb CDF enabled delta extension

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

DuckDelta

DuckDB storage extension that reads and writes Delta tables through DuckDB SQL, using Delta Kernel (delta-kernel-rs) for Delta semantics.

DuckDelta and the delta extension

DuckDB's delta extension (duckdb-delta) reads local and remote Delta tables through delta_scan and ATTACH ... (TYPE delta), supports time travel, and writes only by blind INSERT.

DuckDelta aims to be a Delta extension where change data feed (CDF) is a first-class feature and every storage operation runs as ordinary DuckDB SQL. It is attached with TYPE duckdelta and acts as a DuckDB storage catalog. Writes are held in the DuckDB transaction and committed to the Delta log through Kernel when that transaction commits. It builds its own pinned delta-kernel-rs from third_party/delta_kernel.

Today DuckDelta supports INSERT, and DELETE on tables with deletion vectors, CDF, and row tracking. UPDATE, MERGE INTO and the AT clause are not supported. Paths must be local paths or file:// URIs.

Both extensions link delta-kernel-rs, so their FFI symbols clash if both are linked statically into one libduckdb. Link at most one of them statically.

SQL API

duckdelta_table_version(path) opens the latest Delta snapshot and returns its transaction-log version as UBIGINT:

SELECT duckdelta_table_version('/data/events');

The function returns NULL for a NULL path. Invalid, inaccessible, or unsupported tables raise a Delta Kernel error.

Local setup

The Makefile builds DuckDelta with extension-ci-tools and the duckdb submodule. A build needs VCPKG_TOOLCHAIN_PATH set to a vcpkg toolchain file, because find_package(roaring CONFIG REQUIRED) resolves CRoaring through vcpkg.

Build and test

just build
just build release
just test

just build initializes the duckdb, extension-ci-tools and delta-kernel-rs submodules and runs make. Cargo output is stored under build/<mode>/rust/.

just test runs the unit tests and the SQLLogicTests. Run either suite separately with:

just test-unit
just test-sql

Every command generates the Delta fixtures first.

Unit tests live beside test/sql, one directory per src/ module, and use DuckDB's bundled Catch framework. To rerun after a build without the build check, or to run a single module by its tag, call the binary directly:

cmake --build build/debug --target duckdelta_unittest --parallel 4
build/debug/extension/duckdelta/test/duckdelta_unittest
build/debug/extension/duckdelta/test/duckdelta_unittest '[storage]'

With no argument Catch runs every test in the binary, which is why the recipes pass no filter. A filter matching no test returns failure.

The runner links duckdb_static, duckdelta_extension, the Kernel archive, and, through link_extension_libraries, every statically linked extension. A unit test can therefore open a database and attach a Delta table, which test/storage/test_duckdelta_write_cleanup.cpp does. Which extensions those are follows STATICALLY_LINK_EXTENSIONS in the Makefile.

The unit-test target requires BUILD_UNITTESTS=ON, ENABLE_UNITTEST_CPP_TESTS=ON, and DUCKDELTA_BUILD_CPP_TESTS=ON. Set DUCKDELTA_BUILD_CPP_TESTS=OFF in the CMake cache to omit only this runner. It is also omitted for loadable-only DuckDelta, cross builds, and configuration-only builds. Use just test-sql when the unit-test target is disabled.

Test fixtures

just fixtures builds every table it generates under test/fixtures/: read/ for the tables no test writes to, writable/ for the rest. Both are rebuilt on every run. read/ is committed, so a Kernel bump that changes a table shows up as a diff to review; writable/ is gitignored, because it holds whatever the last test run left behind.

No test names a path under third_party/. Every table taken from the pin is named in KERNEL_FIXTURES in scripts/duckdelta-fixtures.py, so a Kernel bump that renames or drops one fails there. The script also checks each table's schema and partition columns against its own _delta_log before copying it, so a schema change fails there too, before any diff would land under read/.

test/fixtures/ also holds the hand-authored _delta_log fixtures - delta_log_file, empty_string_partition, malformed_log, table_version and transaction_snapshot. Those and read/ are the fixture content in git, and nothing else is.

just fixtures depends on deps and is itself a dependency of the test recipes. read/ is committed, so the read tests pass from a fresh checkout on their own; it is the tables under writable/ that a direct build/<mode>/test/unittest run cannot conjure without just fixtures having run first.

About

duckdb CDF enabled delta extension

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages