Multimode batched evaluation of factorized CC (cost model + placement + dry-run) - #583
Open
evaleev wants to merge 206 commits into
Open
Multimode batched evaluation of factorized CC (cost model + placement + dry-run)#583evaleev wants to merge 206 commits into
evaleev wants to merge 206 commits into
Conversation
CostProfile (peak/flops/exec) over the factorized IR via a zero-data dry-run evaluation, driving the batched cost model's predictions.
Perf-first (DenseTimeSpace) objective with peak_threshold as a ceiling; role-split (contracted/external) batchability; order-aware placement over the combined nest; per-node batch annotations consumed by the evaluator.
External-mode scatter + contracted accumulate; cache scope chain with fall-through; slice-on-use; per-level placement driven by a per-canonical lifetime mask (cross-occurrence meet) unioned with contracted residency; iterative (stack-safe) tree traversal.
Index-space occupancy predicates robust to non-physical spaces; logger; is_valid accepts Power; convention.
The multimode-batched-eval tests had only run under Release/IGNORE, so several Debug-only asserts (and one ASan bug) were masked. Fix them so the suite is green under Debug (SEQUANT_ASSERT_BEHAVIOR=ABORT + AddressSanitizer): - Covariant tensor forms in the synthetic batched-eval tests: contracted indices had been placed in the same bra/ket slot, tripping create_graph's strict-braket invariant. Reorient contractions Einstein-properly and move Hadamard/external indices to the aux slot (test_lifetime_mask, test_eval_ta, test_eval_dryrun). Physical-tensor tests were unaffected (Symm braket). - test_eval_dryrun: a rank-4 CSV composite used a duplicate proto index; use a distinct fourth occ index. - test_eval_ta: rand_tensor_yield now sizes the m (mu~) space; and copy the compared tiles by value in shape_spike_ToT_inner_contraction_to_flat_T (it bound a reference into a temporary Future -> ASan stack-use-after-scope). - test_cache_manager: the batch-axis veto is phase-2 -- a node carrying a batch mode free on its own result is batch-variant (External or Contracted) and correctly refused run-scope caching; update the stale case-3 expectation. - cost_model: seeded_root_peak_batched must admit the seed via the external-role predicate too. The seed is an external mode, so build_context's role filter gates it through is_batchable_external_index; overriding only the contracted-role predicate dropped the seed and tripped the k_seed assert. - Hide ([.]) the all-C60-terms perf-first cost diagnostic: it runs optimize() on every summand (tens of minutes in Debug) and has no correctness checks. - Add a hidden ([.]) water-20 occ-batching overcompute dry-run diagnostic.
order_aware_recompute=false selects the set-keyed DP that ignores the per-batch-block replay recompute, which under-costs every batched schedule and is never the more realistic default. Default it to true in BatchPolicy and CostParams (MPQC and other callers inherit it via optimize()). Pin it false in the two cases that specifically characterize the legacy set-keyed behavior: reconstruct_batched_modes_emits_external_per_node (its own comment documents the order-aware-off emit_external regime) and the C60 objective-determines-factorization case (peak-first forms the fully-sliceable 4-PAO only under the set-keyed peak model; under the realistic resident-scan model the contrast collapses -- peak-first also avoids it and perf-first flops then exceed peak-first because the recompute is charged).
OptimizeOptions::inner_pow and PeakBatchedModel::inner_pow already have no
default (empty + composite indices -> inner_aware_volume throws, so the old
silent mis-sizing fallback cannot recur). But 9 OptimizeOptions{...} designated
initializers omitted inner_pow, which g++ -Wextra -Werror flags as
missing-field-initializers (clang does not, so macOS CI and local clang builds
missed it) -- breaking every Linux Debug job.
Add an explicit .inner_pow = {} (composite-free no-op) at all 9 sites
(optimize.cpp compatibility_opts + 8 in test_optimize). Also finish the
removal that had missed CostParams::inner_pow: drop its stale = {} default and
its stale "sized by idxsz (k=1)" fallback comment so all three inner_pow fields
are uniformly no-default (all 21 CostParams{...} sites already set it).
The static per-node walk in cost_profile() prices each node once, so its
flops/exec are order- and batching-blind and never reflect the per-occ-block
REPLAY recompute the batched evaluator does at runtime -- the reason a dry-run
could not predict occ-batching being slower than aux-only.
Split the reported cost:
- Rename CostProfile::{flops,exec_cost,n_ops} -> model_{flops,exec,n_ops} (the
static DP-model quantities, unchanged).
- Add dryrun_{flops,exec,n_ops}, tallied from the existing Trace::On replay: an
optional CostSink is attached to the shared dry-run CostModel, and every
actual product-op execution (DryRunOps::prod) folds its own SLICED-extent
flops/exec (the same numbers already computed for the per-op OpCost log) into
it. Because a sliced occ-dependent op run N times does ~1/N work per pass, its
sliced-cost sum is work-neutral; only occ-INDEPENDENT work re-executed at full
size once per block inflates -- so dryrun_* isolates exactly the recompute.
The sink is opt-in (nullptr default) and lives on the dry-run CostModel, which
is constructed only inside cost_profile(); eval.hpp / make_evaluator are
untouched, so mpqc's production evaluator path is byte-identical.
water-20 [.][dryrun-water20-overcompute], heavy occ batching: model_flops
ratio 1.0 (flat), dryrun_flops/exec ratio ~1.98, dryrun_n_ops ~55x -- the
recompute is now visible where the model walk saw nothing.
Extend [.][dryrun-water20-overcompute] to three configs -- aux-only, occ+aux
order_aware=false (MPQC production / root-level forest seed), occ+aux
order_aware=true (node-level placement) -- and report the recompute-aware
dryrun_{exec,n_ops} per config plus the OA-true-vs-false ratio.
Diagnoses the water-20 order_aware=true slowdown: under occ batching,
order_aware=true does ~2x the dryrun_exec and ~12x the op-executions of
order_aware=false, and (via its resident-scan peak model reporting higher
peaks) also tips more terms over peak_threshold so occ batching engages where
order_aware=false leaves them un-batched -- a double hit that matches the
observed runtime regression.
Flipping the default to true (previous commit on this branch) turned on the order-aware cost model AND, together with batch_spectator_indices, node-level external placement. On water-20 that path is a runtime regression -- the water-20 dryrun diagnostic measures ~2x the traffic and ~12x the op-executions of the root-level forest seed for the same term -- and, worse, was reported on Owl (job 649250) to produce a WRONG schedule: incorrect PNO-CCSD iteration energies and malformed eval ops. Restore the known-good default (false = legacy set-keyed DP + root-level forest seed) until node-level placement is root-caused and fixed. The two tests that pin order_aware off explicitly keep passing (the pin now merely matches the default).
order_aware_recompute conflated two orthogonal concerns: the order-aware recompute COST MODEL (which factorization the DP selects) and the node-level external-mode EMISSION placement (per-node External stamps vs the root-level forest seed). node_level_placement was defined as order_aware_recompute && batch_spectator_indices, so enabling the more-realistic cost model forced the node-level emission along with it. A water-8 A/B (holding the cost model fixed, toggling only emission) shows the regression is the EMISSION, not the cost model: node-level placement runs ~6x slower (~124 vs ~20 s/iter) and emits ~8x more batch scopes than the root-level seed, because it nests a batch scope at every carrying node and the batched evaluator replays each. The order-aware cost model with root-seed emission is correct and cheap. Node-level placement also produces a wrong residual on water-20 (size-dependent; not reproduced at water-8/he10). Split node_level_placement into its own BatchPolicy/CostParams/CostModel flag (default false), threaded alongside order_aware_recompute. order_aware_recompute now drives only selection; node_level_placement drives only emission. Both default false, so this is behavior-neutral. A gated SEQUANT_NODE_LEVEL_PLACEMENT env knob forces the placement for A/B diagnostics without recompiling a flag through the caller. Tests that engaged node-level placement via order_aware_recompute=true now set node_level_placement=true explicitly; the node-level correctness sweep now drives the emission via its sweep variable.
Now that node-level emission is separately gated by node_level_placement (default off), the order-aware recompute cost model is selection-only and safe to default on: it charges recompute realistically and picks better-batching factorizations, while emission stays the correct, cheap root-level forest seed. Verified no churn across the [optimize] suite (628 assertions) and the batched emission tests. The internal CostModel/oracle-helper member defaults stay false (documented seeded-probe reason); the public BatchPolicy/CostParams path threads the true default through.
…le recompute
Add a batched-evaluation schedule-visualizer pipeline and make avoidable
recompute a first-class per-node output of cost_profile().
- schedule_dump.hpp (new): per-term IR schedule-record emitter
(schedule_ir_json) and a shared cost_op_signature() join key (result index
labels + the sorted operand pair). One definition, used by three producers
so a DAG node, its runtime Build event, and cost_profile's per-node number
all carry the identical key -- no signature is reconstructed downstream.
- eval.hpp: runtime schedule-dump hooks (SCHEDULE_RUN_EVENT / RUN_GROUP),
each internal Build event stamped with the same signature and per-loop
dependent-mode flags, all gated by SEQUANT_SCHED_DUMP (production path
byte-identical when unset).
- CostProfile gains per-node avoidable_nodes {label,count,exec,flops} plus
avoidable_exec / avoidable_ops and avoidable_time(). DryRunOps::prod tallies
each build's necessary = product of block counts of its touched (dependent)
modes, so builds - necessary is the avoidable recompute (a node rebuilt once
per block of a mode it does not touch). The cost model is dense, so necessary
is exact and no empirical correction is needed. avoidable_nodes_from_sink()
is the shared rollup, reused by the schedule-dump test to emit these numbers.
- Consolidate the two dryrun avoidable witnesses (occ-veto, extmode) onto
cost_profile's structural per-node rollup, replacing the BatchGroup/BatchIter
trace-parse string-match reconstruction with a read of cp.avoidable_* (the
structural signature is relabeling-proof; the trace is still parsed only for
the scatter/group Begin markers cost_profile does not expose).
…uckets The per-node avoidable-recompute rollup keyed only by the label signature (result+operand indices), so a node built at many slice sizes landed in one bucket. Because the exec model is roofline-like (nonlinear in slice size), those builds span orders of magnitude (a 257x spread on the C60 external-occ arm); pricing avoidable as count * a single per-build exec then exceeded the whole replay's exec -- the impossible avoidable_time > 100%. Fix: key the sink by the EXTENDED signature (label + the touched modes' realized extents), so every build in a bucket ran at the same slice-context and hence the same roofline cost. Within a cost-homogeneous bucket avoidable_exec = total_exec * (builds - necessary)/builds is exact; the buckets of one DAG node aggregate back to a per-label AvoidableNode (what the visualizer joins on by hash->sig). Verified: buckets are homogeneous (count*last == total*frac per bucket), all arms bounded <= 100% (C60 external-occ 390% -> 0.005%), and the giant term reads 77%, matching an independent dependent-mode analysis (78%). NodeCost now accumulates total_exec/total_flops; min/max/last are kept only for the SEQUANT_AVOIDABLE_DEBUG homogeneity dump.
Redefine the per-value avoidable-recompute metric: it is now measured in FLOPs
against the batching-free (unlimited-memory) ideal -- the arithmetic the batched
replay repeats beyond building each value once at full extent -- rather than in
roofline exec against a within-scheme "necessary" reference.
Two problems with the exec-weighted metric drove this:
- roofline exec is nonlinear in slice size, so one value's differently-sized
builds spanned ~257x; pricing avoidable as count x a single per-build exec
exceeded the whole replay's exec (avoidable_time > 100%). The prior
cost-homogeneous slice-context bucketing removed the >100% but stayed
exec-weighted;
- referencing "necessary = distinct slices the scheme produces" is circular --
it scores an un-hoisted value's per-block rebuilds as necessary, so it cannot
see the recompute hoisting exists to avoid.
FLOPs is linear in extents, hence additive across slices: disjoint per-block
slices that tile a value sum to exactly full_flops (0 avoidable), while a value
rebuilt full per block sums to N*full ((N-1)*full avoidable). So avoidable =
max(0, total_flops - full_flops) per value, bounded in [0, dryrun_flops] by
construction, needs no slice-context bucketing, and answers "what does batching
cost vs. infinite memory". NodeCost drops to {builds, total_flops, full_flops};
CostProfile.avoidable_exec -> avoidable_flops, avoidable_time() = avoidable_flops
/ dryrun_flops.
Witnesses re-baselined (nterms=55, FLOPs): occ-veto 1.8/6.5/15.4%; the extmode
witness shows external-occ (~1.95%) and contracted-occ (~1.97%) essentially
equal -- external-mode batching is NOT a recompute fix on the C60 forest (its
original conclusion, now with honest magnitudes). The [cost_profile] giant reads
77.8%, matching an independent dependent-mode analysis (78%).
The batch-variant caching veto in cache_manager had two disjuncts: (a) a node whose own batched_here() carries a Contracted, batchable mode FREE in its own result, and (b) a non-empty cross-occurrence lifetime mask. Disjunct (a) is structurally dead: a Contracted mode is summed AT the node, so it can never be free in that node's result, and post-role-split a free index is stamped External, never Contracted -- so the condition never holds (the occ-veto test's [veto-reach] probe read 0, structurally, not by forest accident). It only ever guarded a malformed emission. Remove disjunct (a) and the `is_batchable_contracted_index` parameter from the cache_manager factory (the only functional caller, build_dryrun_cache, drops the arg; no production caller passed it), the `CacheConfig::is_batchable_index` field and the cost_profile() overwrite that fed it, and the now-obsolete [veto-reach]/[veto-hazard] probes plus the disjunct-(a) sub-test. Disjunct (b) (the cross-occurrence lifetime-mask veto -- the load-bearing F1 correctness guard) is unchanged. Behavior-preserving: [cache_manager] (200), [lifetime_mask] (76), [dryrun], and [eval] (TA production path, 457) all green. Does NOT touch BatchPolicy::is_batchable_contracted_index (the batching decision) or BatchPolicy::is_batchable_index() (the eval accept union) -- both stay.
Design note reframing batched-eval cache placement as register allocation. Three identities -- value (hash), instance (use-site), cell (a materialized copy serving a subset of instances). A value's instances partition into cells; perfect CSE = one cell/value, no CSE = one cell/instance, and the peak budget chooses the granularity in between (a partial un-CSE / materialization DAG). Cell identity = (value, home-scope, split-index): home-scope is the loop level (the axis batching adds), split-index names a same-scope peak split (the RA live-range-split rename). Placement is register allocation + loop-invariant code motion + rematerialization: hoisting a shared value lengthens its live range (peak) to save recompute; slicing adds partial-hoist granularity. Objective: minimize recompute (the rational, batching-aware reuse count W-1 times build cost) subject to the whole-forest peak profile <= peak_threshold. Peak is a placement (post-CSE, whole-forest) constraint, not a factorizer one; cost_profile()'s replay peak is the detection safety net, and a peak that survives full splitting is factorization-inherent. Includes a prior-art section (rematerialization/checkpointing -- Checkmate; electronic-structure space-time tradeoff -- Cociorva/Sadayappan PLDI 2002; register allocation; pebble games), four worked cases, and open items (group-scoped cache keying, the greedy split move, per-placement footprint, W's fixed point).
O1 (cell keying) resolved as a router + dumb stores, not a wider cache key. Add
§7a "Runtime realization": one value-keyed store per (home-scope, split-index)
-- the cache stays TreeNode-keyed unchanged -- plus an explicit router
{value, use-site} -> (home-scope, split-index) that is the placement pass's
output and replaces the implicit parent_ fall-through search. Reads route via
the map then reuse the EXISTING Enter-stage slicer, (use-scope - home-scope)
INTERSECT carried(N), fed the home scope directly instead of via hops; default
{value} -> (home, 0) is byte-identical. Standardize terminology on "home scope"
(= the code's "lifetime scope" = store scope; consumer's is "use scope"). Update
§4, §9, and O1 accordingly; residual O1 sub-items are the use-site/occurrence id,
the parent_/hops audit, and the naming standardization.
Add §7b: the placement pass as a register-allocation spill loop. Seed = perfect CSE (recompute-minimal, peak-maximal); walk up the recompute axis to walk down peak until peak <= threshold. Objective and constraint are exactly cost_profile()'s avoidable_flops and peak_bytes -- no new measurement. Moves: SHRINK (slice a carried mode a cell holds full -- the existing external-slice / node_level_placement, now driven off the true whole-forest peak) and EVICT (delay/un-hoist an invariant cell held idle, or split a long-lived cell's instances into short-lived groups -- the new CSE-aware move the per-term DP cannot see). Greedy: candidates = cells alive at the binding peak point, prefer free shrinks then max ΔPeak/ΔRecompute (the spill metric), apply, incrementally re-cost, repeat; terminate on fit or on a factorization-inherent peak. Residual sub-items O2a (incremental profile update), O2b (per-move estimator/lookahead), O2c (subsume vs run-after the DP external-slice pass).
Clarify §7b: O2 runs after the per-term min-time factorizer and takes the factorization AND batch-loop assignments (batched_here) as FIXED, deciding only the whole-forest eval/placement strategy (home-scope + router); it never adds, removes, or re-assigns a batch loop. Reframe "shrink" from "slice a carried mode" to "re-home a cell into an EXISTING carried loop" -- a placement choice on the fixed nest, not a batching change; deciding to batch an un-batched mode (adding a loop) is the factorizer's lever. Split the termination boundary into two non-O2 failure modes: factorization-inherent (a single intermediate > budget) vs. re-batch-needed (fixed batching left placement too little room, e.g. a shared cell needing slicing on a mode no single term batched) -- both detected via peak_bytes and fed back, giving the structure factorize+batch -> O2 place -> if infeasible re-batch.
Add §7c. Cell footprint is home-relative: a carried mode is sliced (block extent) iff its fixed batch loop encloses the cell's home, else held full -- the existing moment-aware memsize with home-relative extent overrides, so O2's shrink ΔPeak is just the footprint delta. The peak profile is max weighted-interval overlap: each cell is a [first-use, last-use] interval (from the router's use-sites + the static schedule order) weighted by footprint; peak = max over static points of the sum of live cells' footprints (a sweep line), and the argmax is O2's binding peak point. Because it SUMS co-resident live cells it corrects today's peak_bytes = max(scratch, cache) under-count (a lower bound per §1); the replay stays the oracle (must sum, not max, co-residency). The weighted-interval form updates incrementally under an O2 move (feeds O2a). Residual O3a-c: the sweep structure, the summed- co-residency replay oracle, composite/proto sizing.
Add §7d. Define home_scope(value) = deepest scope enclosing the loops of (sliced_modes ∪ demoted_external_modes). sliced_modes is the cross-occurrence meet (max-reuse upper bound); the demotion fold adds the External batched_here stamps the meet demoted (has_demoted_external) -- occurrences bind them to incompatible blocks, so the value can't be a single full value above those loops and its home must be inside them. The fold is exactly what unifies the current cache-veto-vs-has_demoted_external disagreement into one authority both the cache and the runtime read. Per-block is temporal (one external-loop-homed cell re-instantiated per iteration), so no split-index -- that stays reserved for O2's peak-driven same-scope splits. Structural and computed from the meet before O2, which only lowers homes further for peak; consistent with W (the demoted mode is free tiling). Residual O5a-b: confirm the exact signal / edge cases and the seed router construction. Also tie O6 to §7b's two failure modes.
O4 (W's computation order) is not a fixed point: W is a function of the current placement, well-defined at the home_scope seed and re-costed incrementally per O2 move -- seed-then-refine, subsumed by §7b/§7d. O6 (feedback) scoped to a minimal detect-and-report step (surface the binding cell + failure mode so a schedule fails loudly, not silent OOM), with the re-batch/re-factorize hint as a follow-on that the detect step precedes. All major open items (O1-O6) now designed or resolved; the spec is design-complete.
Phased plan for the placement-as-register-allocation design. Phase 1 (detailed, bite-sized TDD) corrects cost_profile()'s peak_bytes from max(scratch, cache) hwmarks to the instant-resolved co-resident SUM across the scope chain (spec 7c/O3b) -- adds CacheManager current_residency()/chain_residency(), threads the chain sum into note_working_set, simplifies the fold, and re-baselines the documented-RED peak figures from measurement. Phases 2-5 (router+home_scope seed, static peak sweep, the O2 greedy, feedback) are a roadmap, each a future plan. Global constraints: no en-dashes, clang-format, byte-identical perfect-CSE default, replay stays the peak oracle.
…ction The forest-root-combine loop (the only consumer of `perm`) moved into combine_forest_roots (forest_combine.hpp), which recomputes the permute flag itself. The declaration in evaluate_whole_scope was left unused by that extraction; remove it. Behavior-neutral (the variable was never read).
… analysis Add an additive, env-gated diagnostic to measure each cached value's genuine last read, so the water-20 ordered-executor peak can be decomposed into dead-but-retained (pinned past last use) vs genuinely-live residency. - cache_manager.hpp: AccessClock (SEQUANT_UT_ACCESS_CLOCK) -- a global monotonic clock stamped on the two genuine consumer-read paths (access_at, access_at_hops) on hit, with a hash-keyed last-read map. Home entries are pinned (life_c==SIZE_MAX) so their lifetime counter can't reveal last use; this records the real thing. No-op / byte-identical when the gate is off. - test_ordered_executor.cpp: extend the [.][w20-peak-composition] probe to timestamp the peak in the same clock and sum the tier-A home-floor bytes whose last read precedes the peak (the reclaimable-by-eager-release set).
…home-read lifetimes Retire per-batch seeding in the ordered executor for a single read-from-home access discipline, and set each volatile homed value's non-persistent cache life from an EXACT static count of its home reads. - make_batched_scratch gains a read_from_home flag (default off, so whole-scope and forest-descent -- MPQC's batch:whole_scope path -- keep the seeding behavior verbatim). On: a batch-invariant home-resident subnode is read from the parent chain each batch (never seeded), and member ROOTS are cached so a member consuming another member reads it from the scratch instead of re-evaluating it -- removing the last class of non-cached nodes. - ordered_home_reads (ordered_schedule.hpp) computes a homed value's exact home read count from the ordered schedule's realized scopes and the DAG with multiplicity: 1 + sum over direct parents W of prod n_blocks over (build_scope(W) minus home_scope(V)). Wired into both ordered homing sites as the non-persistent life of a volatile homed composite. - CacheManager::resident_in_chain: non-decrementing chain residency probe. Verified on the water-20 witness: predicted == measured exactly (0 mismatches over 121 root-homed composites), build-once holds (worst_ord == 1), realized peak 766.0 GB < the 0.99 TB pin (-22.5%), whole-scope contrast preserved (worst_ws == 7). Guard suite 251/253 (2 pre-existing failures).
…e-accumulating them A batched loop's AccumulateSum/Scatter escape output CONTRACTS the batch axis, so its result is batch-invariant and is homed persistent (kept across cache.reset() between CC iterations). run_ordered_contracted_block re-ran its batch loop every iteration and, for such an output, evaluate_impl resolved the output node from its own home each batch -- handing back the FULL homed value as every batch's 'partial' and summing it N_batches-fold, corrupting the persistent entry into the next iteration (first iteration was correct because the home was still empty). This diverged on water-8 CSV-CCk: iteration 1 matched forest descent exactly, iteration 2 blew up. Fix: at block entry detect outputs already RESIDENT at their home (resident_in_chain) and REUSE them untouched -- skip the re-accumulation. A persistent invariant output is built once and reused across iterations (it survives reset by design); the block still runs for volatile / first-iteration outputs and its loop-local transients. Pre-existing bug, independent of the read-from-home work (reproduces with seeding too). Adds five [eval][ordered-executor] TA regression tests spanning flat/ToT x unbatched/batched-External, an unbatched cross-iteration reset, and the batched Contracted cross-iteration case that reproduced the bug (now passing). Verified on water-8 PNO-CCSD: ordered now converges to the forest-descent reference within batched-contraction numerical noise (residual 1.4e-11).
… trace
The ordered executor emitted no batch-execution marker, unlike the whole-scope
executor (scope_executor.hpp BatchGroup) and the forest batched evaluator
(eval.hpp SCHEDULE_RUN_GROUP), so whether batching actually engaged under
batch:scheduler=ordered was invisible in the eval trace. Add, in
run_ordered_contracted_block right after the batch partition is computed: a
log::printing()-gated BatchGroup "Begin" marker (block axis + batch count) and
a structured SEQUANT_SCHED_DUMP "ORDERED_RUN_BLOCK" line ({kind, mode, blocks,
members}) for scriptable confirmation. Default path (no trace, no env) is
byte-identical.
…r ordered replay + dry==wet equivalence test
Replace BatchPolicy's two coexistence bools (whole_scope_execution,
ordered_schedule_execution) with one BatchScheduler enum {forest_descent,
whole_scope, ordered}, so an ambiguous "!whole_scope" (which wrongly folded
ordered in with forest) can no longer be written. Update every reader in
scope_executor.hpp, cost_profile.hpp, meter.hpp, and the eval tests.
Fix the meter's ordered-replay fidelity bug: it installed the forest batched
custom evaluator for !whole_scope (which included ordered), silently rerouting
ordered builds through the forest evaluator instead of run_ordered_contracted_
block -- diverging from the wet ordered run (which installs no custom evaluator).
Restrict the install to scheduler == forest_descent. MeterReport now carries the
3-way BatchScheduler tag instead of a bool.
Add a dry==wet equivalence test: meter()'s ordered replay peak equals a direct
evaluate_ordered_schedule PeakMonitor peak on the water-20 witness, exactly
(765.97 GB), proving the dry run executes the same schedule the wet run does.
…tion diagnostics Correctness (keep): legality.hpp build_site_of sources batch axes from the DP decision not policy.is_batchable_index(); cost_model.hpp emits no contracted slicing under infinite peak. Migrate two broken pre-existing unit tests. Investigation diagnostics to strip in plan Task 8.
Adds a bool accumulate_in_place() / set_accumulate_in_place() on EvalExpr, next to batched_here()/sliced_modes() (default false, OFF path). binarize() marks it true on every chain Sum node produced when an N-ary Sum is folded into binary Sum nodes: fold_left_to_node (binary_node.hpp) always threads the running accumulator in as the LEFT operand, so for a left-leaning chain (((t1+t2)+t3)+t4) all N-1 binary Sums accumulate their left operand in place. This is groundwork for evaluating CC residual equations as one DAG.
A Sum node that binarize() marked accumulate_in_place() (Task 1: the left-accumulator chain of a folded N-ary Sum) now evaluates via Result::add_inplace() into the left operand's own buffer instead of the allocating Result::sum(), when it is safe to do so: - The right operand is permuted into the accumulator's own layout first (mirroring what the allocating sum() path does for both operands via ann[0]/ann[1]->this_annot), since summands of the original N-ary Sum generally canonicalize to different layouts. Skipped for scalar Sums, which carry no layout and whose Result::permute() is unimplemented. - In-place accumulation is skipped -- falling back to the allocating sum(), despite the mark -- when the left child is a leaf. A leaf's ResultPtr comes straight from the caller-supplied leaf_evaluator, whose provenance (memoized/shared or not) this engine cannot see; mutating it could corrupt an unrelated read elsewhere. This was found empirically: it broke the pre-existing eval_with_btas Summation test, which reads a leaf back out of the same yielder after evaluating a marked Sum that used it as the chain seed. Every OTHER (non-leaf-seeded) Sum in a chain still accumulates in place, since its left child is always a freshly built, evaluation-local buffer. The existing chain_holds() guard against a live shared cache entry is kept as a release-safe SEQUANT_ASSERT. Adds a BTAS-backend unit test verifying, for a 4-term marked chain: the eval trace's per-op mode tally (SumInplace vs Sum) matches which of the 3 marked Sum nodes actually ran in place; every leaf's buffer is untouched after evaluation; and the result matches an unmarked (always-allocating) reference numerically.
The in-place Sum branch (eval.hpp) accumulated its left operand's Result buffer in place whenever the node was marked accumulate_in_place and its left child was a non-leaf, guarding correctness only with a SEQUANT_ASSERT(!chain_holds(f.left)). Under a Release build that assert expands to a no-op, so the guard was elided and the mutation would silently corrupt a value shared across consumers -- e.g. a subexpression CSE'd across two independent roots and homed resident by the ordered executor, read by both. Enforce the safety at runtime instead of via the elided assert: add CacheManager::chain_holds_shared(value) -- true iff a live cache entry with more than one consumer (max_life > 1) still physically holds value -- and gate inplace_eligible on !chain_holds_shared(f.left). A private single-use accumulator is never reported (a transient running total is held by no entry; a single-use CSE entry moves its buffer out on its sole read), so in-place still fires for the common private case; a genuinely shared/resident accumulator falls back to the allocating sum() path. The redundant assert is kept, retargeted to the same predicate. Also factor the ordered schedule walk into detail::run_ordered_schedule_pre_results and add evaluate_ordered_multiroot / evaluate_multiroot (multi-root dispatch returning one result per root, no cross-root summation), with a multiroot driver seam on CacheManager, plus a new test_eval_ordered.cpp whose shared-accumulator case deterministically exercises the in-place safety fix (both roots share a homed a*b; without the guard the shared buffer is corrupted).
CacheManager::chain_holds_shared gated only on max_life_count() > 1, but a PERSISTENT entry (registered via the CacheManager(Iterable&&, PersistencePred) ctor -- the make_batched_scratch path) can have max_life == 1 while entry::access() never drains it: it is resident-forever and shared across the batched replays. Such an entry was reported as NOT shared, so the in-place Sum branch would mutate it -- the exact hazard the guard exists to close, on the batched-scratch replay path. Widen the predicate to persistent() || max_life_count() > 1. This is a strict-safety widening (it only adds persistent held entries to "shared", never removes any) and does not affect the plain forest path, whose CSE entries are non-persistent (test_eval_btas stays 2-of-3 in place). Add direct unit coverage for chain_holds_shared: a persistent max_life==1 entry is reported shared; a non-persistent single-use entry (drained on its sole read) is not.
…iroot
Widen the multiroot layout from a single std::string applied to every
root to a per-root container::svector<std::string> layouts, so the
next consumer (MPQC's combined CC residual call) can feed heterogeneous
roots -- each residual equation carries its own annotation (R1 {a;i},
R2 {ab;ij}, ...) rather than sharing one layout across all roots.
- multiroot_driver_type (cache_manager.hpp): middle parameter widens
from std::string const& layout to
container::svector<std::string> const& layouts.
- evaluate_multiroot (eval.hpp): same widening on its own parameter;
forwards layouts to the driver; throws std::logic_error if
layouts.size() != roots.size().
- evaluate_ordered_multiroot (ordered_executor.hpp): same widening;
SEQUANT_ASSERT(layouts.size() == root_nodes.size()); the per-root
combine loop now passes layouts[i] to combine_forest_roots instead
of a single shared layout.
- test_eval_ordered.cpp: updated call sites to pass
svector<std::string>(roots.size()) of empty strings for the existing
scalar-root tests (byte-identical to the old single-empty-layout;
ScalarEvalExpr has no tensor backend, so a real distinguishable-
layout permute isn't exercisable here), the installed driver
closure's signature to match, and a new size-mismatch guard test.
…ated BuildMeter counter) Strip BuildMeter::life_event()/trace_hash() (the per-value [life-seq-dbg]/[life-seq] tracing, gated on SEQUANT_UT_TRACE_HASH) and their two call sites in CacheManager::access_at()/store(), along with the now-dead local captures those calls consumed. Also strip the env-gated SEQUANT_UT_CELL_CHECK/SEQUANT_UT_CELL_DUMP diagnostic block in evaluate_ordered_schedule (ordered_executor.hpp). BuildMeter's build-COUNTING (enabled()/on_build()/on_read()/Reporter, gated on SEQUANT_UT_BUILD_METER) is unaffected and stays opt-in; it proved the single-DAG multiroot eval fix and remains useful going forward.
Compare a Product's children as an unordered pair so a contraction emitted as (X,Y) in one term and (Y,X) in another folds to a single cache entry, matching the operand-order-independent node hash and canonical connectivity graph. The recursive child comparison is retained -- the graph encodes only the immediate two factors' connectivity, not each factor's recursive build, so two products with the same immediate graph but different sub-values stay distinct. Non-Product nodes keep the ordered child comparison.
The un-split residual/energy is kept as a single in-place Sum-tree with one
node per summand, so its left spine is as deep as the number of terms --
thousands for a UCC BCH energy expansion. Two tree traversals still recursed
down that spine and overflowed the call stack (observed on the
h2o+-ucck-2-bch2e4 open-shell UCC input):
- FullBinaryNode::size(): recursive left().size() + right().size().
Rewritten to an explicit-stack node count, mirroring the already-iterative
destructor / deep_copy. It also ran O(N)-deep on every use, e.g. each
equality comparison's size check.
- TreeNodeEqualityComparator::operator(): recursed on the left child.
The left-child descent is now unwound into a loop; right children (single
summands) and Product operands are bounded in depth and stay recursive.
This is a faithful transcription of the recursive comparison -- same
per-node checks, same ordered child compare, same commutative-Product
unordered match -- so dedup semantics (incl. batch-index-specific CSE) are
unchanged; only the deep spine is iterated.
…ip pruned BuildSteps Three coupled fixes to the ordered (DAG) executor's memory/lifetime behavior, bringing its footprint in line with forest descent (w8 PNO-CCSD DAG peak footprint 26.8 -> 18.6 GB, matching forest; energy byte-identical): - value_results no longer retains a parallel ResultPtr to every produced value -- a reference living PAST the cache's own drain (~14 GB over-hold on w8). A forest root is now homed with +1 life for its combine read and read back from the cache; non-roots were always cache-held. One holder, exact counts. - ensure_home_slot consults the shared cache's persistence classification (CacheManager::entry_is_persistent -- non-volatile AND a volatile DIRECT consumer) instead of recomputing !vol, which over-enrolled non-volatile values with no volatile consumer and pinned them across all iterations. - A per-iteration "needed" gate (BFS from the volatile roots through not-currently-alive nodes; a persistent cache hit does not propagate need to its children) prunes BuildSteps whose consumers are all cached, so un-persisting those values no longer forces a wasteful rebuild each iteration -- mirroring forest descent's stop-at-cache-hits. Also: a forest leaf is no longer emitted as a standalone root BuildStep (it is an input fetched on demand by its consumers, not a computed value), matching forest descent; RichSchedule::ValueCell gains is_leaf.
…ff profile The base.total_flops constant and the aux+occ peak comment were pinned in 6137461 and drifted as ~88 later commits landed (commutative-Product node dedup 6ecb6d5, the ordered-schedule / peak-monitoring stack). Two witness numbers moved over that span, neither from a change in what the witness measures: - base.total_flops (perfect-CSE floor): 1.5798408944778614e16 -> 1.5793702764908846e16 (-0.03%). - aux+occ gate-off peak: ~563 GB -> ~146 GB (a real improvement from the intervening factorization / peak work). Update the total_flops constant, refresh the peak comment, and tighten the peak CHECK from < 600 to < 200 GB so it again guards just above the achieved value. Measured legs are otherwise unchanged (unbatched 50.8 TB / 0% avoidable, aux-only 18.6 TB / 0%, aux+occ 146 GB / 75.8% avoidable; role stamps Contracted-occ=0, External-occ=244).
… CSV false positive)
is_valid's Sum branch requires all summands to share the same external indices,
extracted by get_unique_indices -- which walks only a tensor's own bra/ket/aux
slots and never descends into proto-indices. In a CSV (proto-indexed) residual
an occ index lives both as a standalone slot (g{mu~;i;K}, t{...;i}) and inside a
composite virtual's proto-list (a<i,j>); the slot-only count is then not
invariant across summands, so is_valid spuriously reports "Inconsistent external
indices in sum" on a perfectly valid CSV residual. On an MPQC_ASSERT_ABORT build
MPQC's MPQC_ASSERT(is_valid(e)) then aborts every CSV-CCk run (all ranks,
deterministically) right after equation processing; on THROW/IGNORE builds it
did not fire, which is why it went unnoticed.
Fix: when the reference summand carries proto-indices, validate consistency with
a proto-aware external-index set -- an index is external iff its total
occurrence count (slot appearances plus appearances as a proto-index of a
composite) is odd. Real CSV residual terms always carry the occ externals
standalone (in a t-amplitude ket or an integral slot) in addition to the protos,
so this set is invariant across summands. Non-proto-indexed expressions are
unaffected: they keep the original per-group get_unique_indices comparison
verbatim.
Adds a fast regression test that deserializes the real CSV doubles residual and
asserts is_valid accepts it (it reported "Inconsistent external indices in sum"
before this change).
…ped CSV-CCk batching) opt_pure_product keys each summand's per-contraction-node batch annotations (term_batch_axes) by that optimized summand's Product pointer. optimize_impl's Sum path then reassembles the summands into a new Sum -- and, under reorder, opt::reorder appends them via Sum::append, which CLONES each summand -- so the final summands have new pointers and the keyed pre-clone summands are destroyed when optimize_impl returns. A consumer that binarizes the whole reassembled Sum in one call and looks the annotation up by the final Sum pointer (as MPQC's CCk does now that the residual is one in-place Sum-tree per equation, not a per-summand forest) therefore finds nothing: every batch annotation is silently dropped, no index is sliced, and over-budget intermediates materialize whole -- e.g. the 2.6 TB unsliced 1-PNO/1-PAO DF integral in water-20 PNO-CCSD, an OOM on a 773 GB node. (The per-summand dry-run path is unaffected: it consumes each annotation immediately, before the pointer dies.) Fix: in optimize_impl's Sum path, gather the per-summand node_batch_axes in the FINAL summand order (the flattened cluster order opt::reorder emits; identity without reorder) into one whole-tree vector -- binarize walks the Sum-tree in that same left-first post-order, one entry per contraction node, so it stays aligned with binarize's node counter -- store it under the final Sum pointer, and drop the now-unreachable per-summand entries. The whole-Sum lookup then returns the correct concatenated annotation. Adds a regression test asserting term_batch_axes is keyed by the reassembled Sum (with real axes) on the actual CSV doubles residual.
…lel summand optimization) optimize_impl optimizes a Sum's summands with sequant::for_each (std::execution::par_unseq / a manual thread pool), so opt_pure_product runs concurrently across summands. Its insert into the shared term_batch_axes map is a data race: std::unordered_map concurrent inserts are UB even on distinct keys (a rehash re-links every bucket). The map is then corrupted, and the whole-Sum re-key added in the previous commit reads a wrong-sized node_batch_axes, tripping binarize's `node_counter == node_batch_axes.size()` assertion -- a nondeterministic, thread-count-dependent SIGABRT. It surfaced on water-20 PNO-CCSD on Owl (GCC+TBB, 9 threads/rank) after batching finally engaged; it stays hidden where par_unseq falls back to sequential (libc++) or the manual pool's timing happens not to collide. The parallel-invariant note above the for_each covers the per-task clones and the deferred Index::label reads but never covered this shared map write. Guard just the insert with a mutex; the heavy DP stays parallel. (The race predates the re-key commit -- opt_pure_product always wrote the map in the parallel branch -- but was inert while the whole-Sum lookup silently missed every entry.) Extends the term_batch_axes regression test to also binarize the residual with the concatenated node_batch_axes, so a size/alignment mismatch is caught in-test.
…ix non-Unity build) These test TUs reference sequant::mbpt::Spin (add_pao_spaces(isr, Spin::any), etc.) but did not include the header that defines it -- add_pao_spaces takes an IndexSpace::QuantumNumbers, so convention.hpp does not drag mbpt::Spin in. The CMAKE_UNITY_BUILD grouping happened to place these files with a TU that did include it, so a unity build compiled; a non-unity build (e.g. the Owl GCC config) fails with "sequant::mbpt::Spin has not been declared" on test_ordered_schedule.cpp and would fail the same way on the others. Include the header directly in each TU. Verified with a full CMAKE_UNITY_BUILD=OFF build of unit_tests-sequant.
… node-count off-by-one)
single_term_opt(Product const&) early-returns for products with fewer than 3
factors (no DP is needed for a single contraction), but it left out_axes empty.
A 2-tensor product (e.g. the R1 singles term f{mu~;i} * C{a<i>;mu~}) is one
contraction, so binarize builds one contraction node -- but with zero node_axes
entries the caller's concatenated node_batch_axes comes up one short of what
binarize emits, tripping binarize's node_counter == node_batch_axes.size()
assertion. It aborted water-20 PNO-CCSD on Owl (SEQUANT_ASSERT_BEHAVIOR=ABORT)
in build_residual_artifacts, on the R1 equation; a THROW/IGNORE build (incl. the
local test build) never fired it, and the doubles residual happens to have no
bare 2-tensor summand, so R2-only tests missed it.
Fix: the early return now emits (#tensor factors - 1) empty NodeBatchAnnotation
entries, mirroring run_single_term_opt_axes's nt==1 (zero) / nt==2 (one) cases
-- a scalar*tensor product still yields zero (no contraction). Adds a regression
test built on the exact water-20 R1 summand that reproduces the off-by-one via
the same DenseTimeSpaceBatched optimize + head-pinned binarize path MPQC uses
(node_axes count must equal binarize's contraction-node count).
occurrence_key flattens a node's subtree leaves into one TensorNetwork to compute a routing key, which is only well-defined for a tensorial (contraction) subtree. A Sum node unions its summands' leaves -- each reusing the same dummy labels independently -- so an index connects to >1 bra/ket slot and create_graph's strict-braket invariant (correctly) rejects the malformed network deep in canonicalization (the water-20 PNO-CCSD crash on all ranks). Add a precondition in occurrence_key that fails fast on a Sum with a clear message, and reject a non-Product aggregate up front in TensorNetworkV3(const Expr&): its subexpressions are iterated as the factors of a single network, which is meaningful only for a Product.
… multi-physical-label aux blocks Two changes to carry water-20-scale CSV-CCk through the ordered (multiroot) executor with aux batching: 1. Router-consult moved()-gate (eval.hpp). The remat pass keys occurrence_key only for the values it actually moved; gate the eval-time consult (both the forest-descent Enter stage and the hoist path) on router->moved(hash) before computing a key. occurrence_key is thus never handed a Sum node (never moved), and it is byte-identical -- a non-moved node route()-missed anyway. 2. DAG-space -> node-physical-mode map (member_axis.hpp, ordered_executor.hpp). build_ordered_schedule buckets batch members by axis TYPE (space), so one block may co-evaluate members that bind that space under different physical labels (K_1, K_2). Map the block's canonical axis to each member's OWN physical label -- for the members list / make_batched_scratch, the scatter destination, and the mode_batches tiling source -- reusing member_contracted_axis / member_external_axis (hoisted out of scope_executor.hpp into member_axis.hpp, since scope_executor.hpp includes ordered_executor.hpp). slice-on-use space-maps its axis through a new CacheManager::space_mapped_slicing flag that ONLY the ordered executor sets; the whole-scope evaluator keeps exact-match precision (default off, byte-identical). This replaces the ordered_axis_label_mismatch assert, which rejected the now-handled multi-label case. Regression test [occurrence-key] (test_eval_dryrun.cpp) reproduces the whole chain -- occurrence_key precondition, moved-gate, and multi-label remap -- on the water-20 residual equations with no MPQC/HF/PNO-MP2 run. The rank-0 scalar observable is excluded from the dry-run forest, as MPQC does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Multimode batched evaluation of factorized coupled-cluster equations
Adds cost-model-driven multimode batching to the SeQuant evaluator so
large-system CSV/PNO-CC residuals can be evaluated without forming their
largest transients whole. Six squashed commits (dry-run backend, optimizer,
evaluator, supporting core, tests, docs).
What it does
-> scattered into disjoint slices) and contracted (DF aux, summed ->
accumulated). Loops nest external-outside-contracted.
DenseTimeSpace): minimizes flops withpeak_thresholdas a ceiling; role-split (contracted/external) batchability;order-aware placement over the combined nest.
(a cached intermediate fetched from an outer scope is sliced to the current
block) decouples correctness from placement; per-level placement driven by
a per-canonical lifetime mask (cross-occurrence proto-aware meet) unioned
with contracted residency; iterative (stack-safe) tree traversal.
C60 PNO-CCSD dry run (55-term residual, aux K@256, occ@8, 100 GB budget)
The DP selects the same factorization regardless of what is batchable
(flops are unchanged); batching only slices modes to lower the peak. The
roofline-time column moves because a giant intermediate executed whole is
memory-bound (
machine_balance x traffic) but compute-bound when sliced -- thecache-blocking win of the same schedule, not a cheaper one. Recompute overhead
(
avoidable_time) is 1.8% -> 6.5% -> 39.8% as slicing gets more aggressive.Validation
[eval]449,[lifetime_mask]76,[optimize]628 assertions green;OFF (order-blind) path byte-identical.
events) matches unbatched to < 1e-9, within the 1e-7 precision, no aborts.
Follow-ups (non-blocking, from the final review): dedup the proto-expansion
helper; add a real-forest hidden-tag hash-regression test; revisit the
stamp_lifetime_masksconst_cast.