#426 - L1 taxonomy: classify cancellations and cascade-only TCPStore logs

[continue-revolution]

Five category descriptions in l1/categories.json. No decision values change; no code changes.

Each was found by reading a production misclassification back to the description that caused it.

What changed

cat	problem	fix
12	680 chars, longest of 38 (median 142), and the only one in prohibitive voice. No clause for "the initiating rank's output is absent", so the model declined instead of classifying.	Say what to do when TCPStore is the only surface: pick it, with reduced confidence. Keep the guard that a visible upstream fault wins.
1	Read, in full, "no application fault" — false on its face, since a kill leaves plenty of application-shaped noise.	Give it both surfaces a cancellation actually leaves: the bare slurmstepd marker, and progress-then-teardown. Name the cat 2 discriminator.
28	Names "TypeError/AttributeError on None", which matches interpreter-teardown noise verbatim.	Exclude anything raised during shutdown: a deterministic code fault recurs on unchanged retry, so it fails on the first attempt, not after 3390 iterations.
34	Pinned rank 608 and the GPTDataset index-load path.	Described as a class; discriminating signatures kept.
35	Pinned ckpt 27600.	Same. (Its row indices were already removed on main by 3e96f3a, which also covered cat 29.)

Why it mattered

Hand-cancelled jobs were being recommended STOP. Their logs hold 751 SIGTERMs, 847 Exception ignored and healthy iterations to the last line before the signal; the grounded primary failure was Python's own weakref finalizer losing a module global during teardown — which Python labels Exception ignored precisely because it did not affect execution.

Separately, cascade-only TCPStore logs returned category_id: 0 ("none matched") at confidence 0, so the alert told the reader nothing about a job that had cleanly checkpointed and then lost its store.

Validation

• 26 live Nemotron logs, re-analyzed on an isolated instance: 5 changes, all accounted for. Three 0 → 12 (intended), one 0 → 1 (a correction — that log has 6180 SIGTERMs and 3 TCPStore lines), one 26 ↔ 32 shown to be model variance by repeat trials on identical input, both STOP.
• 13 labeled dataset cases covering every edited category, with positive and negative controls: 0 regressions, 1 improvement. Config errors still STOP, node failures still cat 2, labeled TCPStore cases still cat 12.
• Seven incidents inspected individually. Five cancellations corrected STOP → RESTART. 4188567 correctly left at STOP/cat 17 — its kill followed a real MissingEpochIndex dataset fault (7104 occurrences, zero completed iterations), and the broadened cat 1 did not claim it.

Note for reviewers

An earlier revision of cat 1 described only the progress-then-teardown shape and broke the canonical case — two labeled cases whose entire log is *** STEP <id> ON <node> CANCELLED AT <time> *** drifted to cat 2. That is the same over-fitting this PR removes from cat 12, and it was caught only because the control set covered every edited category; both categories are RESTART, so headline accuracy was unchanged. Worth keeping that shape of control for future taxonomy edits.

Two follow-ups not addressed here: cats 26 and 32 are not reliably distinguishable (demonstrated twice), and cats 1 and 3 both have a claim on a log containing only a cancellation marker.

🤖 Generated with Claude Code

[greptile-apps]

Confidence Score: 4/5

[Low impact] Updates taxonomy descriptions for log classification.

The PR should not merge until category 28 stops excluding repeatable code errors merely because they occur after successful training.

Findings

1.  Late code errors lose STOP ▶

Summary

Updates descriptions for scheduler cancellations, downstream termination signals, TCPStore failures, and native crashes. Removes decision wording from several descriptions without changing their stored decisions.

• Cancellation descriptions now cover both bare scheduler markers and progress followed by signal-driven teardown.
• TCPStore descriptions distinguish setup failures from downstream errors after training progress.
• continue-revolution explicitly deferred the overlap between categories 26 and 32, and between categories 1 and 3 for cancellation-only logs.

_{Reviews (9) · Last reviewed commit: "Merge branch 'main' into chengsongz/taxo..." · Reviewed by Greptile}

[greptile-apps]

Late code errors lose STOP

Category 28 now excludes errors “appearing only after sustained successful iterations” and says deterministic faults fail early. A reproducible TypeError or AttributeError can first occur in a later evaluation step or a data-dependent model branch, without any shutdown or external kill. The new description excludes that error from its correct STOP category.

If the category-driven restart override is enabled and the model selects category 1 instead, the job can restart only to hit the same error. Limit the exclusion to errors shown to come from interpreter shutdown; successful earlier iterations do not rule out a deterministic workload bug.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[continue-revolution]

Generated by human --

This PR is mainly to fix:

• Patterned TypeError failure caused by slurm cancellation. Previously go to launch config error category (c28) and STOP, now go to slurm cancellation category (c1) / off-site termination category (c3) and RESTART
• Patterned TCPStore failure caused by outside termination. Previously go to no match category (c0), now go to off-site termination category (c3), action unaffected
• Remove any actions / policy in category description, and remove any data index leak from category description