DuckDB storage extension that reads and writes Delta tables through DuckDB SQL, using Delta Kernel (delta-kernel-rs) for Delta semantics.
DuckDB's delta extension (duckdb-delta) reads local and remote Delta tables through delta_scan and
ATTACH ... (TYPE delta), supports time travel, and writes only by blind INSERT.
DuckDelta aims to be a Delta extension where change data feed (CDF) is a first-class feature and every
storage operation runs as ordinary DuckDB SQL. It is attached with TYPE duckdelta and acts as a DuckDB
storage catalog. Writes are held in the DuckDB transaction and committed to the Delta log through Kernel
when that transaction commits. It builds its own pinned delta-kernel-rs from third_party/delta_kernel.
Today DuckDelta supports INSERT, and DELETE on tables with deletion vectors, CDF, and row tracking.
UPDATE, MERGE INTO and the AT clause are not supported. Paths must be local paths or file:// URIs.
Both extensions link delta-kernel-rs, so their FFI symbols clash if both are linked statically into one
libduckdb. Link at most one of them statically.
duckdelta_table_version(path) opens the latest Delta snapshot and returns its
transaction-log version as UBIGINT:
SELECT duckdelta_table_version('/data/events');
The function returns NULL for a NULL path. Invalid, inaccessible, or
unsupported tables raise a Delta Kernel error.
The Makefile builds DuckDelta with extension-ci-tools and the duckdb submodule. A build needs
VCPKG_TOOLCHAIN_PATH set to a vcpkg toolchain file, because find_package(roaring CONFIG REQUIRED)
resolves CRoaring through vcpkg.
just build
just build release
just test
just build initializes the duckdb, extension-ci-tools and delta-kernel-rs submodules and runs make. Cargo
output is stored under build/<mode>/rust/.
just test runs the unit tests and the SQLLogicTests. Run either suite separately with:
just test-unit
just test-sql
Every command generates the Delta fixtures first.
Unit tests live beside test/sql, one directory per src/ module, and use DuckDB's bundled Catch
framework. To rerun after a build without the build check, or to run a single module by its tag, call
the binary directly:
cmake --build build/debug --target duckdelta_unittest --parallel 4
build/debug/extension/duckdelta/test/duckdelta_unittest
build/debug/extension/duckdelta/test/duckdelta_unittest '[storage]'
With no argument Catch runs every test in the binary, which is why the recipes pass no filter. A filter matching no test returns failure.
The runner links duckdb_static, duckdelta_extension, the Kernel archive, and, through
link_extension_libraries, every statically linked extension. A unit test can therefore open a
database and attach a Delta table, which test/storage/test_duckdelta_write_cleanup.cpp does. Which
extensions those are follows STATICALLY_LINK_EXTENSIONS in the Makefile.
The unit-test target requires BUILD_UNITTESTS=ON, ENABLE_UNITTEST_CPP_TESTS=ON, and
DUCKDELTA_BUILD_CPP_TESTS=ON. Set DUCKDELTA_BUILD_CPP_TESTS=OFF in the CMake cache to omit only this
runner. It is also omitted for loadable-only DuckDelta, cross builds, and configuration-only builds.
Use just test-sql when the unit-test target is disabled.
just fixtures builds every table it generates under test/fixtures/: read/ for the tables no test
writes to, writable/ for the rest. Both are rebuilt on every run. read/ is committed, so a Kernel
bump that changes a table shows up as a diff to review; writable/ is gitignored, because it holds
whatever the last test run left behind.
No test names a path under third_party/. Every table taken from the pin is named in KERNEL_FIXTURES
in scripts/duckdelta-fixtures.py, so a Kernel bump that renames or drops one fails there. The script
also checks each table's schema and partition columns against its own _delta_log before copying it,
so a schema change fails there too, before any diff would land under read/.
test/fixtures/ also holds the hand-authored _delta_log fixtures - delta_log_file,
empty_string_partition, malformed_log, table_version and transaction_snapshot. Those and read/
are the fixture content in git, and nothing else is.
just fixtures depends on deps and is itself a dependency of the test recipes. read/ is committed,
so the read tests pass from a fresh checkout on their own; it is the tables under writable/ that a
direct build/<mode>/test/unittest run cannot conjure without just fixtures having run first.