Repository navigation
dbt Core 2.0 parity: General adapter behaviour #712
Description
Sub-issue of #660 — feature-parity tracking for the ClickHouse adapter in dbt Core 2.0 / Fusion versus the Python dbt-clickhouse adapter.
Covers adapter-wide behaviour: capability probes (EXCHANGE, lightweight deletes), server version checks, column schema introspection, macro sync with upstream, S3 source support, connection-level behaviour and usage tracking. Metadata-based source freshness is a dbt Core 2.0-only capability and is tracked in #736.
Status legend: ✅ implemented — merged and working on dbt-core 2.0 origin/main · 🟡 partial — partly working on origin/main (Notes say what works vs fails) · 🚧 in progress — implemented in an open PR (linked in Notes) · 🔴 not started — not working anywhere · ⚪️ future work — explicitly deferred, not being solved now (also listed in the future-work sub-issue) · Combined status "A-🚧->B": the feature is at A on origin/main today; an open PR is driving it to B (🟡 partially working / ✅ fully working)
| Feature | Subfeatures / details | Status | Notes (PR) |
|---|---|---|---|
| General adapter behaviour | (overall) | 🟡 | Merged: version probe, DESCRIBE introspection, TLS, non-experimental registration, S3 (dbt-labs/dbt#16082), lw-delete probe (dbt-labs/dbt#16090), EXCHANGE probe (dbt-labs/dbt#16200), 0.1.1 driver (dbt-labs/dbt#16185), dbt/<version> User-Agent (dbt-labs/dbt#16156). Missing: usage tracking (dbt first in the User-Agent), query comment, query_id, view replace fallback without renameat2 |
| Server version checks | SELECT version() probe + adapter.is_before_version / is_at_or_after_version |
✅ | dbt-labs/dbt#16079: once-per-process probe, compare_versions mirrors util.py; used by the 26.6 insert_overwrite gate |
adapter.format_columns / get_column_schema_from_query |
DESCRIBE TABLE (sql) with the model's query_settings; ADBC/Arrow reports all columns Nullable, so the server-typed schema replaces the driver schema |
✅ | dbt-labs/dbt#16079 (fixes dbt-labs/dbt#14581) |
| TLS / secure connections | secure profile flag, HTTPS/TLS driver support |
✅ | Driver 0.1.0-a2 for TLS (dbt-labs/dbt#10391); secure registered with port/scheme autodetection |
| Adapter registration / onboarding | ClickHouse listed as a non-experimental adapter; dbt init flow |
✅ | Non-experimental (closes dbt-labs/dbt#15410); dbt init improvements (dbt-labs/dbt#10558) |
| Lightweight-delete capability probe | use_lw_deletes profile flag + settings/permission checks |
✅ | Probe in dbt-labs/dbt#16090; allow_nondeterministic_mutations sent for the disabled-but-changeable case in dbt-labs/dbt#16156 (adbc_clickhouse#70) |
ADBC driver: literal ? in SQL |
A literal ? must reach the server verbatim (the 0.1.0 driver treated it as a bind param, forcing \x3F/?? escapes into seed.sql, adapters.sql and materialized_view.sql) |
✅ | adbc_clickhouse 0.1.1 (adbc_clickhouse#53) via dbt-labs/dbt#16185 sends a literal ? verbatim; dbt-labs/dbt#16156 retired seed.sql's ?? escape. The \x3F regex escapes in adapters.sql/materialized_view.sql are harmless server-side escapes and stay until retired together with the v1 macro tree |
| S3 table function | adapter.s3source_clause (clickhouse_s3source(...) via project vars), incl. external_id/role_arn extra credentials |
✅ | dbt-labs/dbt#16082 with impl.py parity |
| UUID support in ADBC / Arrow | Driver-side: ADBC driver 0.1.0 (adbc_clickhouse#63); server-side: Arrow output of UUID columns requires ClickHouse 26.7+ |
✅ | Driver fix released (adbc_clickhouse#63). The server-side 26.7+ requirement is a Known limitation in #660; no adapter work |
| EXCHANGE capability check | Probe once per process: db-engine check + real CREATE/EXCHANGE/DROP scratch test; check_exchange: false skips; relation.can_exchange stamped on cached relations and used by the table / incremental / snapshot / materialized-view swaps |
✅ | dbt-labs/dbt#16200. Rebuilds and full refreshes of tables, incremental swaps, snapshot upserts and MV target rebuilds run EXCHANGE TABLES on Atomic/Replicated/Shared databases and fall back to rename elsewhere |
| Macro sync to upstream | Byte-identical macros; only documented Fusion divergences allowed | 🟡 | Most of the tree is byte-identical (dbt-labs/dbt#16023, #16079, #16082, #16200, #16153, #16552). Still differing on main: adapters.sql (get_columns_in_query return form), persist_docs.sql (extra validate_doc_columns call), view.sql (no supports_atomic_exchange fallback), seed.sql. Missing entirely: adapters/apply_grants.sql, materializations/clone.sql, utils/datatypes.sql. Deliberate divergences: adapters.sql clickhouse__check_schema_exists, seed.sql's literal-VALUES insert (the format CSV path is deferred, #731), table.sql on_cluster_clause (asks adapter.should_on_cluster(relation=…) instead of reading relation.should_on_cluster, so relations built before the connection exists still get the clause) |
| Connection profile keys parity | cluster, cluster_mode, use_lw_deletes, check_exchange, local_suffix, local_db_prefix, allow_automatic_deduplication, custom_settings, database_engine, sync_request_timeout, compress_block_size, secure, … |
🟡 | Consumed on main: secure, custom_settings (driver options, dbt-labs/dbt#16156), check_exchange (EXCHANGE probe, dbt-labs/dbt#16200), use_lw_deletes (lw-deletes probe, dbt-labs/dbt#16090), allow_automatic_deduplication (dedup default, dbt-labs/dbt#16079), database_engine/cluster_mode (connection settings + DDL), cluster (ON CLUSTER DDL and the cluster-aware catalog, dbt-labs/dbt#16552), local_suffix/local_db_prefix (distributed macros). Missing: the HTTP-client keys verify, client_cert/client_cert_key, connect_timeout, send_receive_timeout, sync_request_timeout, compress_block_size parse but are inert — the ADBC driver exposes no TLS-verification, client-certificate, timeout or compression options, so these need driver work first |
| Usage tracking | dbt identifies itself first in the HTTP User-Agent (system.query_log.http_user_agent), as dbt/<version> … |
🟡 | v1 sends dbt-adapters/x dbt-clickhouse/y clickhouse-connect/z (lv:py/…; os:…; os_user:…). Merged (dbt-labs/dbt#16156): dbt/<version> via the driver's clickhouse.client.product_info option — DDL/INSERT statements log dbt/2.0.0-rc.1 adbc_clickhouse/0.1.1 clickhouse-rs/0.15.2 (…), but Arrow reads (SELECT/DESCRIBE) log clickhouse-ext-arrow/0.1.0 dbt/2.0.0-rc.1 … because clickhouse-ext-arrow tags the query last and clickhouse-rs prints the last-added product first. Missing: dbt/… always first — reported as ClickHouse/clickhouse-rs#469; the fix has to land there, then ship in a clickhouse-rs release, then in an adbc_clickhouse release, and only then can dbt Core 2.0 bump its driver pin (no dbt-side code change expected). Extra key/value pairs in the parenthesised comment are not possible either (clickhouse-rs hardcodes that section) |
View replace fallback without renameat2 |
adapter.supports_atomic_exchange(); view.sql falls back to drop + create when the filesystem lacks renameat2 (NFS), upstream #684 |
🔴 | Only the method name is registered on main (no implementation) and view.sql has no fallback, so CREATE OR REPLACE VIEW fails with UNSUPPORTED_METHOD on such filesystems (test_view_no_renameat2) |
| Query comment handling | query-comment: null honored; rendered macro comments re-validated |
🔴 | 2 integration tests fail (NullQueryComments, MacroInvalidQueryComments). Engine-generic fix |
adapter_response.query_id |
Engine-assigned clickhouse.client.query_id surfaced into run results/logs |
🔴 | 1 integration test fails |
| Relation cache population timing | The relation cache must be filled once per schema, before the first node runs (dbt-core populate_adapter_cache), so every materialization sees the same existing_relation state (mvs_pointing_to_it, can_exchange, …) |
🔴 | Engine behaviour, not adapter code: v2 lists a schema lazily when the first model asks for it (and concurrent misses list it again, the last one overwriting), so the state a model sees depends on thread timing; dbt run and dbt run --select <model> can disagree and TestMVWithExplicitTargetForcesOnSchemaChangeFailInTable is flaky on v2. Filed as dbt-labs/dbt#16454 (2026-09-25); depends on dbt-labs/dbt#16429 landing first so ClickHouse fills the cache at all. The Fusion distribution has a pre-run "downloading relations" warm-up that the open dbt-oss build lacks (only defer and dev-clone warm the cache there), so dbt-oss is the more exposed build |
dbt debug target handling |
Bad target should surface a connection-failure report, not a profile-resolution error | ⚪️ | Deferred (#731). An unknown --target errors out before the dbt debug report is printed |
dbt run --sample (sample mode) |
Models' inputs filtered by the --sample window |
⚪️ | Deferred (#731). A sampled model returns 0 rows on ClickHouse |
| Duplicate hook definitions | A node defining both post-hook and post_hook (or the pre- pair) must fail compilation |
⚪️ | Deferred (#731). v2 keeps one of the two hooks silently instead of failing |
| Port dbt-clickhouse 1.10.1 features | Bring the features added in v1 1.10.1 to 2.X | ⚪️ | Deferred (#731); not itemized yet |
| Port dbt-clickhouse 1.10.2 features | Bring the features added in v1 1.10.2 to 2.X | ⚪️ | Deferred (#731); not itemized yet |
Activity