Skip to content

fix(learning): stop discarding runs and undercounting CPU - #78

Merged
Dunky13 merged 2 commits into
mainfrom
dunky13/fix-learning-measurement
Nov 5, 2026
Merged

Dunky13 merged 2 commits into
mainfrom
dunky13/fix-learning-measurement

Conversation

@Dunky13

@Dunky13 Dunky13 commented Nov 5, 2026

Copy link
Copy Markdown
Owner

Summary

Most measured runs recorded as incomplete, so no workload variant could earn evidence for overlap or exact predictions. On Apple Silicon, CPU also read about 42 times too low.

 sample a job
-  any process appears or exits mid-sample  -> sample fails
+  complete when every process that lasted through it was read
-  one failed sample                        -> run incomplete
+  failed sample                            -> counts only if the job is sampled again
+                                              never advances the last observation
 job reported finished, receipt still held
-  receipt present                          -> run lost
-  samples keep updating the run
+  samples stay out of its evidence, admission still reserves what it holds
 finalize
-  sample after the recorded end            -> run lost
+  tolerated up to the sampling interval
-  duration = last sample - start
+  duration = recorded end - start
 Darwin CPU
-  ri_user_time + ri_system_time read as ns
+  converted from Mach ticks with the timebase

This is the first of three PRs. Per-repository grouping and the project and operation browse hierarchy follow separately.

Evidence

Same four jobs on a private daemon and scheduler, including one with constant worker churn:

  • Before (main): 0 of 4 runs successful, peak CPU 0.01 cores, 0 of 12 complete samples on the churn job.
    After: 4 of 4 runs successful, CPU 0.36 to 0.38 cores, 11 of 12 samples complete.
  • Before: TestResourceDarwinCPUTimeIsNanoseconds reported 0.006 s against the kernel's 0.251 s.
    After: passes.

scripts/test-go-runtime.sh core-1 and core-2 and scripts/lint.sh pass locally on the code commit. Mutation checks confirm the new finalization, restart and admission tests fail when each behaviour is removed. The full scripts/validate.sh, including its vulnerability check and tooling shards, did not run locally. Intel relies on hosted CI.

Independent reviews (Claude Opus 5.5) found five regressions during this work, all fixed here: a stall hidden by a failed final sample, a good late sample discarding a run, post-finish samples changing evidence, a job first seen finished stopping every scheduler tick, and admission ignoring what a finished job holds.

Merge Danger

Door: two-way

Runs recorded by this version stay in history after a revert, but the schema does not change.

Blast Radius: scheduling

  • Once out of shadow mode, Adaptive refuses more heavy jobs: with correct CPU units, a peak above about 64% of cores predicts over budget and needs Start now.
  • Existing CPU history is not migrated, so it under-reserves until each variant runs again.
  • Workers that start and exit between samples stay invisible, and these runs can now become evidence, so CPU totals remain lower bounds.
  • Jobs under roughly 4 to 6 seconds still record as incomplete, which predates this PR.

See docs/handoffs/learning-measurement.md.

🤖 Generated with Claude Code

Dunky13 and others added 2 commits November 5, 2026 15:18
Most measured runs never counted as successful, so no workload variant
could earn the evidence that adaptive overlap and exact predictions need.
Every sample failed when a worker started or exited while it was being
read, and one failed sample marked the whole run incomplete. The last
sample of nearly every job also failed by racing the job's own exit, and
a job seen as finished before its receipt was cleaned up lost its run.

A sample is now complete when it read every process that lasted through
it. A failed sample counts only if the job is sampled again, and it never
advances the last observation, so a real gap still fails finalization.
Samples of a job the queue reports as finished add nothing to its
evidence, and recorded duration is the job's actual execution time.

Native CPU counters are Mach ticks, about 41.67 ns on Apple Silicon, but
were treated as nanoseconds. They are now converted with the timebase.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marc Went <marc@went.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marc Went <marc@went.io>
@Dunky13
Dunky13 merged commit 678ec61 into main Nov 5, 2026
20 checks passed
@Dunky13
Dunky13 deleted the dunky13/fix-learning-measurement branch November 5, 2026 21:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant