Skip to content

feat(unstable): opt-in Charon CDF-writer integration - #723

Draft
nithinb wants to merge 3 commits into
masterfrom
nithin/CharonUploadQueue
Draft

nithinb wants to merge 3 commits into
masterfrom
nithin/CharonUploadQueue

Conversation

@nithinb

@nithinb nithinb commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

Why?

Charon is a new CDF-writer service (POC, changing contract). Instead of extractors writing timeseries straight to CDF, they can register their task config once and stream data through Charon, which decorates/routes/shapes items (Kuiper expressions, regex space routing) and forwards to CDF with the caller's own bearer token. We want extractors built on this library to opt into Charon without disturbing the existing direct-to-CDF path, and without committing the POC's unstable wire contract to the stable public API.

What?

Opt-in support, entirely under Cognite.Extractor.Utils.Unstable.Charon. No stable type changes behavior or signature. Disabled by default (cdf-writer.enabled: false) → direct-to-CDF path is byte-for-byte unchanged.

  • ICharonClient / CharonClient — typed client for /cdfwriter/setup and /cdfwriter/payload; the single isolation seam for the wire format. Reuses the existing bearer-token HTTP flow; logs only status/counts, never tokens or item bodies.
  • CharonWriter — façade to declare the task set (CharonTask.SpaceRouting / .Custom), register it at startup (SetupAsync), and create upload queues.
  • CharonUploadQueue / CharonDatapointsUploadQueue — IUploadQueue<T> writers that chunk each task's items to CDF limits and map results to QueueUploadResult<T>.
  • OrderedStringMap — preserves space_routing mapping order (first regex match wins).
  • Config / DI — cdf-writer section on ConnectionConfig, DestinationUtilsUnstable.AddCharonWriter, gated in the runtime. Schema + docs updated; version → 1.46.0.

Key decisions:

  1. Fallback = config switch only. On Charon 5xx/unreachable the upload fails like any CDF error; flip enabled: false to use the direct path.
  2. Setup runs once at startup. Charon's store is in-memory, so a Charon restart mid-run causes payload failures until the extractor restarts.
  3. Typed builders + raw JsonElement escape hatch for task registration and item bodies.
  4. Datapoints = one reading per payload item (no pivot); throughput caveat documented.

Carry-forward: regex anchoring for space routing is not yet frozen Charon-side (full-match vs substring) — client sends patterns verbatim, noted in builder docs.

How?

Extractor authors:

var writer = Provider.GetRequiredService<CharonWriter>();
writer.RegisterTask(CharonTask.SpaceRouting(
    "route-ts", CharonDestinations.TimeSeriesCreate,
    new[] { new KeyValuePair<string,string>("abcd.*", "space-1") },
    defaultSpace: "space-default"));
await writer.SetupAsync(GetExtractorVersion(), token);

var dps = writer.CreateDatapointsQueue("dps", TimeSpan.FromSeconds(5), 10000, logger);
dps.Enqueue("ts-external-id", new Datapoint(DateTime.UtcNow, 42.0));

Config (connection.yml):

cdf-writer:
  enabled: true
  base-url: "https://charon.example.com"

Testing

19 new tests: wire-contract (camelCase + mapping-order), client (200/207/422/401/5xx), queues (chunking, datapoint expansion, DI). Full Unstable suite 154/154 green; dotnet build of the solution clean.

Note: test project targets net9.0; this machine has net8/net10 runtimes, so tests ran with DOTNET_ROLL_FORWARD=LatestMajor.

Nithin Bodanapu added 3 commits October 10, 2026 12:48
Route timeseries create/update/datapoints through the Charon CDF-writer
service instead of directly to CDF, behind an opt-in config switch. All new
types live under Cognite.Extractor.Utils.Unstable.Charon, since Charon is a
POC with a changing contract. The wire format is isolated behind ICharonClient.

- CharonClient: typed /cdfwriter/setup and /cdfwriter/payload client, reusing
  the existing bearer-token HTTP flow; logs only status/counts, never tokens
  or item bodies.
- CharonWriter: façade to declare the task set, register it at startup, and
  create upload queues.
- CharonUploadQueue / CharonDatapointsUploadQueue: IUploadQueue<T> writers that
  chunk to CDF limits and map results to QueueUploadResult<T>.
- OrderedStringMap: preserves space_routing mapping order (first match wins).
- Config: opt-in cdf-writer section (disabled by default -> direct-to-CDF
  path, unchanged); wired through DestinationUtilsUnstable.AddCharonWriter and
  gated in the runtime. Schema updated, version bumped to 1.46.0.
- Wire-contract: camelCase field names, insertion-ordered mapping
  serialization, error-body and payload-response parsing.
- Client: setup 200/422, payload 200/207/422, 401 and 5xx.
- Queues: payload body shape, chunking to CDF limits, datapoint
  one-item-per-reading expansion, result mapping, and opt-in DI registration.
Document the opt-in cdf-writer config, task registration, write path,
and the fallback/limits/POC caveats. Register it in the tutorials TOC.
@nithinb
nithinb requested a review from a team as a code owner October 10, 2026 07:37
@nithinb
nithinb marked this pull request as draft October 13, 2026 04:13

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant