You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The failure is reproducible across test runs and only appears on machines with the
MNIST raw data downloaded. The test self-skips otherwise (@pytest.mark.slow + data
check), which is only caught if CI is run with MNIST downloaded.
Importance
Reproducibility. For any harness built on this framework: RNG state consumed at import time silently shifts every
subsequent seeded draw (i.e. weight initialization, data shuffling and sampling, augmentation/drift
simulation parameters, dropout, and stochastic detector behavior), so identically seeded
experiments can produce different metrics, drift timing, and losses depending on nothing more than
import order or whether a module was already cached.
Most likely happened when lazy imports of examples was introduced.
Root cause (found by Claude Fable 5.1; verbatim)
git bisect between 992691c (PR #73, where tests/references/mnist_first_drift_ewc_losses.csv was recorded) and current main
identifies b95da75 (PR #82, "Make import statements for model classes conditional
based on the dataset name") as the first bad commit. The loss values it produces are
bit-for-bit identical to today's — no later commit changed the numerics further.
PR #82 didn't change any math. The divergence comes from an RNG-ordering interaction:
examples/mnist/model.py defines MNIST_CNN.__init__(self, cfg, model: nn.Module = Cnn()). The default Cnn() is
constructed at module import time, and its weight initialization consumes draws from
torch's global RNG. (Verified: seeding torch to 1337 and importing examples.mnist.model changes the next torch.rand(1) from 0.0783… to 0.7181…; no
other module in the import chain consumes RNG.)
After Make import statements for model classes conditional based on the dataset name #82, the import happens lazily inside get_example(), after seeding. The Cnn() construction now burns RNG draws post-seed, shifting every subsequent draw
(affine drift parameters, dataloader shuffling, dropout) and therefore all logged CL
losses. The random weights themselves are irrelevant — they're overwritten by mnist.pth; only the RNG side effect matters.
Suggested Work
The MNIST test is an end-to-end numerical regression test. Fix the example's RNG ordering, then regenerate the
reference CSV.
Make sure other templates do not have import-time RNG side effects.
Assert that importing each example model module leaves torch.get_rng_state() (and NumPy/Python RNG state) unchanged, so no example performs RNG-consuming work at import time.
Symptom
tests/test_valiadation_tests.py::test_mnist_first_drift_losses_match_referencefails deterministically:The failure is reproducible across test runs and only appears on machines with the
MNIST raw data downloaded. The test self-skips otherwise (
@pytest.mark.slow+ datacheck), which is only caught if CI is run with MNIST downloaded.
Importance
Reproducibility. For any harness built on this framework: RNG state consumed at import time silently shifts every
subsequent seeded draw (i.e. weight initialization, data shuffling and sampling, augmentation/drift
simulation parameters, dropout, and stochastic detector behavior), so identically seeded
experiments can produce different metrics, drift timing, and losses depending on nothing more than
import order or whether a module was already cached.
Most likely happened when lazy imports of examples was introduced.
Root cause (found by Claude Fable 5.1; verbatim)
git bisectbetween992691c(PR #73, wheretests/references/mnist_first_drift_ewc_losses.csvwas recorded) and currentmainidentifies
b95da75(PR #82, "Make import statements for model classes conditionalbased on the dataset name") as the first bad commit. The loss values it produces are
bit-for-bit identical to today's — no later commit changed the numerics further.
PR #82 didn't change any math. The divergence comes from an RNG-ordering interaction:
examples/mnist/model.pydefinesMNIST_CNN.__init__(self, cfg, model: nn.Module = Cnn()). The defaultCnn()isconstructed at module import time, and its weight initialization consumes draws from
torch's global RNG. (Verified: seeding torch to 1337 and importing
examples.mnist.modelchanges the nexttorch.rand(1)from 0.0783… to 0.7181…; noother module in the import chain consumes RNG.)
examples/utils.pyimported all example model modules eagerly, so thishappened before the test called
torch.manual_seed(seed)— the seeded stream wasuntouched, giving the reference values.
get_example(), after seeding. TheCnn()construction now burns RNG draws post-seed, shifting every subsequent draw(affine drift parameters, dataloader shuffling, dropout) and therefore all logged CL
losses. The random weights themselves are irrelevant — they're overwritten by
mnist.pth; only the RNG side effect matters.Suggested Work
reference CSV.