feat(workflow-tengo): make ColumnUniversalId a first-class column reference - #1780
Draft
AStaroverov wants to merge 6 commits into
Draft
feat(workflow-tengo): make ColumnUniversalId a first-class column reference#1780AStaroverov wants to merge 6 commits into
AStaroverov wants to merge 6 commits into
Conversation
The model mints a recursive column id — a bare leaf, or a Filtered /
Overridden / Discovered wrapper around one — and the workflow so far
recognized only the legacy `{source, axisFilters}` form, whose `source`
is a map rather than an id string.
Add `:pframes.column-id`, the decoder those ids need: classification,
layer peeling down to the terminal leaf, and the per-layer spec math
mirroring `spec/ids.ts`, `spec/filtered_column.ts` and
`spec/overridden.ts` on the model side.
`poolKey` is the load-bearing piece. A rich id keys on itself, never on
its leaf, so two projections of one physical column — or one column
reached by two linker routes — stay distinct entries. The legacy form
keeps keying on its `source`, and marker fields are therefore tested
before that duck-test.
One deliberate divergence from the model side: an `axesSpec` patch whose
index is past the end of the spec panics instead of appending an axis.
The workflow registers specs against data that already exists, and an
axis with nothing behind it is a spec the engine cannot satisfy.
Everything here is pure over decoded JSON, so it is covered offline.
No caller yet.
A linker join inner-joins its linker with the secondary side and then projects out the linker's one-side axes, so a column reached through one does not land on its own axes. Nothing workflow-side could compute that. Port the derivation: `splitAxes` is a union-find over `parentAxes`, which is declared data on the axis rather than something inferred from domains, so no axis normalization has to come with it. `linkerSides` requires exactly two components, as the engine does, and takes the one holding axis 0 as the one side. `linkerJoinAxesSpec` composes them — linker axes lead, matching the engine's integration order, and a hit axis the route never touches survives. Component ordering follows the engine's: components are ordered by their lowest member index, so the numbering agrees on both sides. Also export `A_IS_LINKER_COLUMN`. `pt.p.linkerJoin` reads it through `pframesSpec` but it was never exported, so the builder's first caller would have hit a strict-map panic. It has none today, which is why this went unnoticed.
The collector and the unmarshaller each carried their own id sniffing, and the two had to agree or a column registered under one key would be looked up under another. Route both through `columnId.poolKey` so they cannot drift, and accept the rich shapes on the way. A rich id registers under itself and resolves its terminal leaf by ref; a Discovered id additionally registers every hop in its route, since a linker is a column in its own right. Reads apply the layers the id carries: axis filters slice the data as before, spec overrides patch the spec, and a Discovered layer passes through — the linker chain enables co-indexing without remapping the hit's axes, and the engine needs the hit's real axes to resolve the join. `getQueryEntry` is where a route stops being metadata and becomes a join: every hop folds into a nested `linkerJoin` around the hit, ready for `pt.p._rawQueryEntry`. It re-asserts `pl7.app/isLinkerColumn` itself, because reaching the engine without it silently degrades the join to an inner join rather than failing. Fixes a latent disagreement: `getAxesSpec` read raw pool specs while `getSpec` applied axis filters, so a filtered id could contribute axes its own spec no longer had. `frameFromColumnBundle` calls the former before the latter, so this was reachable. Both now report the same thing, and for a discovered id that means the post-join axes.
… route `frameFromColumnBundle` decided how to add a column by looking for an `axisFilters` field on the decoded key. That misreads every new shape: a Filtered id nested under an Overridden one has no field there, and a Discovered id's route was ignored entirely, so the column joined on the wrong axes. Ask the id instead. A route becomes a real `linkerJoin` through `getQueryEntry`, a Filtered id is sliced by ptabler as before, and a bare or spec-patched leaf is a plain column. Pool keys that are not ids at all — multi-result element keys, caller-supplied query keys — keep the original path untouched.
`EnrichmentRef` and `ColumnDiscoveredKey` say the same thing — a column plus the linker route to it — in two spellings, and `tableBuilder` understood only the older one. The older one is also strictly narrower: its hit and every hop must be bare global ids, and its only step type is `linker`. So normalize it into the other rather than teaching the builder two mechanisms. `fromEnrichmentRef` converts, `normalizeDiscovery` reduces either spelling to one shape, and a single resolution path follows. The marker check stays the first dispatch branch, so existing callers take the same route through the builder they always did. The reverse conversion is lossy and deliberately absent: a discovered key whose hit or hop carries a projection has no v1 representation. Mark the type deprecated, with no removal date. `tableBuilder` refuses an id carrying spec overrides or axis filters rather than dropping them — it resolves specs as futures, so there is nothing to patch or slice at that point, and a column bundle is the right home for those. A discovered id as a primary is refused too: a primary is the trunk, and a discovered column is reached from one.
Records where each id layer is realised, why a rich id keys on itself in the pool, how a discovered column's frame axes are derived, and the four places the workflow deliberately panics instead of silently diverging from the model side — including why `queriesQualifications` cannot be carried at all.
🦋 Changeset detectedLatest commit: 3c27ae2 The changes in this PR will be included in the next version bump. This PR includes changesets to release 24 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| // A route can be resolved here because every hop is a column to fetch. A | ||
| // projection cannot: specs arrive as unresolved futures, so there is nothing | ||
| // to patch or slice at this point. The bundle path handles those. | ||
| unwrapped := columnId.unwrap(key) |
Contributor
There was a problem hiding this comment.
When an EnrichmentRef contains a supported pre-resolved {spec, data} hit, normalizeDiscovery passes the converted key to columnId.unwrap, which descends into that map and rejects it as a selector before resolveColumn can handle it, causing tableBuilder.build() to panic during rendering.
Knowledge Base Used: workflow-tengo: the block workflow authoring SDK
Prompt To Fix With AI
This is a comment left during a code review.
Path: sdk/workflow-tengo/src/pframes/table-builder.lib.tengo
Line: 126
Comment:
**Resolved EnrichmentRef hits panic**
When an `EnrichmentRef` contains a supported pre-resolved `{spec, data}` hit, `normalizeDiscovery` passes the converted key to `columnId.unwrap`, which descends into that map and rejects it as a selector before `resolveColumn` can handle it, causing `tableBuilder.build()` to panic during rendering.
**Knowledge Base Used:** [workflow-tengo: the block workflow authoring SDK](https://app.greptile.com/milaboratories/-/custom-context/knowledge-base/milaboratory/platforma/-/docs/workflow-tengo.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1780 +/- ##
==========================================
+ Coverage 49.57% 53.04% +3.46%
==========================================
Files 70 375 +305
Lines 3786 20070 +16284
Branches 969 4445 +3476
==========================================
+ Hits 1877 10646 +8769
- Misses 1640 8119 +6479
- Partials 269 1305 +1036 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
AStaroverov
marked this pull request as draft
August 18, 2026 17:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes
ColumnUniversalIda first-class column reference in@platforma-sdk/workflow-tengo. A block can pass any id the model mints — bareleaf,
ColumnFilteredId,ColumnOverriddenId,ColumnDiscoveredId— intocreatePBundleBuilder().addSingle(...),pt.frameFromColumnBundleortableBuilder, and the workflow resolves it, reconstructs its effective spec,and builds the linker join a discovered id describes.
Why
The model migrated to the new column access API, so args now carry
ColumnUniversalIdwhere they used to carry a leafPObjectId. The workflowrecognized only the legacy
{source, axisFilters}form — whosesourceis amap, not an id string — so every new shape fell through to a branch that pushed
it into
bquery.anchoredQueryin an invalid form.The route mattered most. Previously identity travelled on the wire and the
workflow re-derived the linker path itself, heuristically. A
ColumnDiscoveredIdcarries the route, so the workflow can build the joinexactly as chosen instead of guessing.
How each layer is realised
bquery.resolveon the terminal{__isRef, …}Filteredslice-data, orpt.p.slicedColumnin a frameOverriddenspecOverridenodeDiscoveredbundle.getQueryEntry→bquery.buildQuery→pt.p._rawQueryEntryTwo design points worth reviewing closely:
Pool keys. A rich id registers and reads under the id itself, never its
leaf. Two projections of one physical column, or one column reached by two
routes, are different columns to a consumer — this is the workflow's counterpart
to
rebrandLeafId. It also makes the registration key and the read keyidentical by construction, so the collector and the unmarshaller cannot drift.
The legacy
FilteredPColumnIdkeeps keying on itssource, so marker fields aretested before that duck-test.
Axes after a linker join. The engine projects out the linker's one-side axes,
so a discovered column does not land on the hit's own axes.
pSpec.linkerSidesderives the two sides from the linker's spec by splitting
axesSpecintoparent-connected components —
parentAxesis declared data on the axis, so noaxis normalization had to be ported. A hit axis the route never touches survives,
which a blanket empty
axesSpecwould lose.EnrichmentRef becomes derived
EnrichmentRefis the strictly narrower spelling of the same idea, so itnormalizes into
ColumnDiscoveredKey(columnId.fromEnrichmentRef) andtableBuilderresolves both spellings through one path. The marker check staysthe first dispatch branch. The type is deprecated; no removal date. The reverse
conversion is lossy and deliberately absent.
Deliberate panics, not silent divergence
axesSpecpatch past the end of the spec — the model appends an axis, theworkflow cannot introduce one that carries no data.
sliceAxesnode inside thejoin, which the query builder cannot emit. Filter after the join.
tableBuilder— specs there are unresolvedfutures. Use a column bundle.
tableBuilderprimary — a primary is the trunk, adiscovered column is reached from one.
queriesQualificationsis not carried: its keys arePObjectIds of externalprimary columns, and the workflow addresses primaries by
PlRefand frame key.Documented rather than dropped quietly.
Latent defects fixed on the way
bundle.getAxesSpecread raw pool specs whilegetSpecapplied axis filters,so a filtered id could contribute axes its own spec no longer had.
frameFromColumnBundlecalls the former before the latter, so this wasreachable today.
pSpec.A_IS_LINKER_COLUMNwas read bypt.p.linkerJoinbut never exported —the builder's first caller would have hit a strict-map panic. It has none
today, which is why it went unnoticed.
Back-compat
Untouched:
workflow/bquery.lib.tengo(including its closed query schema),query-anchored.tpl.tengo,slice-data.tpl.tengo,build-query.lib.tengo,build-table.tpl.tengo,process-column*,xsv*. All public signaturessurvive, including
addQuery(queryKey, query)'s inverted argument order and thedeprecated
addById.table-builder.test.tengo's existing cases pass unchanged.Testing
pl-tengo checkclean andpl-tengo testgreen at every commit (232 → 248tests, 0 failures). 39 new offline tests cover the id decoding, the spec math per
layer, the pool-key derivations existing bundles depend on, the component split,
and the
EnrichmentRefnormalization.Not covered here: an end-to-end run against a live backend. The intended check is
blocks/antibody-tcr-lead-selectionon a project with upstream clustering, afterdropping the
extractPObjectIdflattening in itsmodel/src/util.ts— thatblock's hand-rolled linker matching (
findMatchingLinkerIndex) becomesunnecessary once the route is honoured.
Review order
Six commits, each independently building and tested:
column-id.lib.tengo— the decoder, no callerslinkerSides— plus the missing exportEnrichmentRefderived, deprecationGreptile Summary
The PR makes logical column identifiers directly consumable by workflow bundles, frames, and
tableBuilder, including execution of discovered linker routes. It also derives legacyEnrichmentRefhandling from the new representation, adds linker-axis projection utilities, and documents the expanded workflow contract.ColumnUniversalId— the logical recursive identifier for a column, including leaf, filtered, overridden, and discovered forms; this PR adds Tengo-side decoding, registration, specification reconstruction, and execution support.ColumnDiscoveredId— a column identifier containing a hit and linker route; this PR turns its route into nested executable linker joins and derives contributed frame axes from those joins.ColumnFilteredId— a logical column projection with pinned axis values; this PR reconstructs its effective specification and slices its data when constructing frames.ColumnOverriddenId— a logical column reference carrying specification metadata patches; this PR applies those patches when reading a column bundle.EnrichmentRef— the legacy hit-plus-linker-route reference, including support for already-resolved prerun columns; this PR deprecates it and normalizes it toward the discovered-ID representation, but the new normalization currently breaks its resolved-hit case.PColumnBundle— the resolved pool of column specifications and data; this PR unifies registration/read key derivation and adds query-entry and effective-axis APIs.linkerJoin— the query operation that traverses a linker from its one side to its many side; this PR builds it from discovery paths and projects the linker's one-side axes from frame output.poolKey— the identity under which a bundle stores and retrieves a column; rich identifiers now key by the complete logical ID while legacy filtered identifiers retain source-based keying.Confidence Score: 4/5
The PR should not merge until tableBuilder preserves legacy EnrichmentRefs containing already-resolved prerun columns.
The new unconditional unwrap of normalized EnrichmentRefs turns a documented and previously supported resolved-column input into a render-time panic.
Files Needing Attention: sdk/workflow-tengo/src/pframes/table-builder.lib.tengo; sdk/workflow-tengo/src/pframes/column-id.lib.tengo
Important Files Changed
Flowchart
%%{init: {'theme': 'neutral'}}%% flowchart LR ID[ColumnUniversalId] --> Decode[column-id decode and unwrap] Decode --> Leaf[Resolve terminal leaf] Decode --> Filters[Apply axis filters] Decode --> Overrides[Apply spec overrides] Decode --> Route[Read discovery path] Route --> Linkers[Resolve linker columns] Linkers --> Query[Build nested linkerJoin query] Leaf --> Bundle[PColumnBundle] Filters --> Bundle Overrides --> Bundle Query --> Frame[PTabler frame entry] Bundle --> Frame Legacy[EnrichmentRef] --> Normalize[Normalize to discovered key] Normalize --> DecodePrompt To Fix All With AI
Reviews (1): Last reviewed commit: "docs(workflow-tengo): document the workf..." | Re-trigger Greptile
Context used: