Skip to content

import-time RNG consumption in MNIST_CNN's default Cnn() argument invalidated the numerical reference #129

Description

@rz4

Symptom

tests/test_valiadation_tests.py::test_mnist_first_drift_losses_match_reference fails deterministically:

step actual reference
2 6.850990295410156 7.514564037322998
3 5.589588880538940 5.548082828521729

The failure is reproducible across test runs and only appears on machines with the
MNIST raw data downloaded. The test self-skips otherwise (@pytest.mark.slow + data
check), which is only caught if CI is run with MNIST downloaded.

Importance

Reproducibility. For any harness built on this framework: RNG state consumed at import time silently shifts every
subsequent seeded draw (i.e. weight initialization, data shuffling and sampling, augmentation/drift
simulation parameters, dropout, and stochastic detector behavior), so identically seeded
experiments can produce different metrics, drift timing, and losses depending on nothing more than
import order or whether a module was already cached.

Most likely happened when lazy imports of examples was introduced.

Root cause (found by Claude Fable 5.1; verbatim)

git bisect between 992691c (PR #73, where
tests/references/mnist_first_drift_ewc_losses.csv was recorded) and current main
identifies b95da75 (PR #82, "Make import statements for model classes conditional
based on the dataset name")
as the first bad commit. The loss values it produces are
bit-for-bit identical to today's — no later commit changed the numerics further.

PR #82 didn't change any math. The divergence comes from an RNG-ordering interaction:

  • examples/mnist/model.py defines
    MNIST_CNN.__init__(self, cfg, model: nn.Module = Cnn()). The default Cnn() is
    constructed at module import time, and its weight initialization consumes draws from
    torch's global RNG. (Verified: seeding torch to 1337 and importing
    examples.mnist.model changes the next torch.rand(1) from 0.0783… to 0.7181…; no
    other module in the import chain consumes RNG.)
  • Before Make import statements for model classes conditional based on the dataset name #82, examples/utils.py imported all example model modules eagerly, so this
    happened before the test called torch.manual_seed(seed) — the seeded stream was
    untouched, giving the reference values.
  • After Make import statements for model classes conditional based on the dataset name #82, the import happens lazily inside get_example(), after seeding. The
    Cnn() construction now burns RNG draws post-seed, shifting every subsequent draw
    (affine drift parameters, dataloader shuffling, dropout) and therefore all logged CL
    losses. The random weights themselves are irrelevant — they're overwritten by
    mnist.pth; only the RNG side effect matters.

Suggested Work

  • The MNIST test is an end-to-end numerical regression test. Fix the example's RNG ordering, then regenerate the
    reference CSV.
  • Make sure other templates do not have import-time RNG side effects.
  • Assert that importing each example model module leaves torch.get_rng_state() (and NumPy/Python RNG state) unchanged, so no example performs RNG-consuming work at import time.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions