Repository navigation
Conversation
miketynes
commented
Sep 30, 2026
miketynes
commented
Collaborator
- add support for fairchem
- Use fairchem as reference calculator
- Demonstrate mace training to fairchem
…exceptions An unhandled exception during dynamics execution (e.g. numpy.linalg.LinAlgError from ASE's MTK-NPT barostat) previously propagated out of the DynamicsRunner's run loop and crashed the agent, silently stalling the trajectory while the Slurm job kept reporting RUNNING. Now such failures are caught, logged, and recorded via mark_trajectory_failed before a clean agent shutdown.
MAD-vs-timestep scatter now colors points by which model_version produced each frame (via a helper that mirrors get_trajectory_atoms's chunk iteration/dedup exactly). New "UQ vs error" section plots DBTrainingFrame.calibration_uq against calibration_error (the same pair the Controller uses for real threshold calibration), log-log, colored by model_version_sampled_from.
model_version is a small integer category, not a continuous quantity, so a viridis colorbar was misleading. Both the MAD plot and the UQ-vs-error plot now scatter one group per model_version with a proper legend.
Pulls the first-ever frame and the last-ever recorded frame directly (not via get_trajectory_atoms, which only returns passed chunks) so a trajectory stuck on a failed/crashed final attempt still gets a real end-state comparison instead of silently falling back to its last passed chunk.
No per-member bootstrap provenance is persisted (Trainer.train_model draws indices with an unseeded RNG, held only in memory), so this characterizes the round-1 training pool directly instead: every one of its 200 frames has a labeled-atoms min interatomic distance under 1 A (98.5% under 0.5 A), i.e. essentially collided structures. A Monte Carlo simulation of the actual bootstrap procedure on that real pool shows meaningfully uneven per-member exposure to the worst frames arises from resampling variance alone, even though every member draws from the same uniformly pathological pool.
* Mark trajectory failed and shut down DynamicsRunner on dynamics task exceptions An unhandled exception during dynamics execution (e.g. numpy.linalg.LinAlgError from ASE's MTK-NPT barostat) previously propagated out of the DynamicsRunner's run loop and crashed the agent, silently stalling the trajectory while the Slurm job kept reporting RUNNING. Now such failures are caught, logged, and recorded via mark_trajectory_failed before a clean agent shutdown. * Add model-version coloring to MAD plot and a UQ-vs-error plot MAD-vs-timestep scatter now colors points by which model_version produced each frame (via a helper that mirrors get_trajectory_atoms's chunk iteration/dedup exactly). New "UQ vs error" section plots DBTrainingFrame.calibration_uq against calibration_error (the same pair the Controller uses for real threshold calibration), log-log, colored by model_version_sampled_from. * Use discrete per-model-version legends instead of a continuous colormap model_version is a small integer category, not a continuous quantity, so a viridis colorbar was misleading. Both the MAD plot and the UQ-vs-error plot now scatter one group per model_version with a proper legend. * Add cell-volume start-vs-end plot per trajectory Pulls the first-ever frame and the last-ever recorded frame directly (not via get_trajectory_atoms, which only returns passed chunks) so a trajectory stuck on a failed/crashed final attempt still gets a real end-state comparison instead of silently falling back to its last passed chunk. * Add notebook investigating ensemble member 2's training divergence No per-member bootstrap provenance is persisted (Trainer.train_model draws indices with an unseeded RNG, held only in memory), so this characterizes the round-1 training pool directly instead: every one of its 200 frames has a labeled-atoms min interatomic distance under 1 A (98.5% under 0.5 A), i.e. essentially collided structures. A Monte Carlo simulation of the actual bootstrap procedure on that real pool shows meaningfully uneven per-member exposure to the worst frames arises from resampling variance alone, even though every member draws from the same uniformly pathological pool. * Guard against undetected training-pool/ensemble divergence Default burn-in to 1 round so a real retrain (and real inter-member divergence) always happens before the UQ audit is load-bearing -- previously an ensemble of clone-initialized members had ~zero disagreement at model_version=0, so audits only ever caught anything by the unrelated random-fail perturbation. Filter physically-absurd frames (min interatomic distance below a configurable threshold) out of the training pool before they reach the trainer, and detect per-member training divergence (non-finite validation loss) with a retry-on-fresh-bootstrap loop instead of silently deploying diverged weights. Add gradient-norm clipping and a NaN/Inf loss guard in MACEInterface.train to keep a single bad batch from corrupting a member outright.
…91) * Persist NPT/MTKNPT barostat/thermostat state across chunk boundaries Every chunk of a trajectory ran as a separate executor task, and advance_dynamics constructed a brand-new ASE NPT/MTKNPT integrator each time. ASE zeroes the barostat/thermostat extended-system state (cell/barostat momenta, Nose-Hoover chain variables, or eta/zeta for classic NPT) unconditionally in __init__, and cascade never persisted or restored it, so the barostat/thermostat effectively restarted from rest at every chunk boundary even though atomic positions/momenta/cell carried over correctly. This produced a visible discontinuity in atomic motion rate at each chunk seam (e.g. every 1000 steps), independent of any model-version change. DynamicsRunner now carries the integrator's extended-system state across chunks (mirroring how self.atoms already carries over, including only advancing on a passed audit so a retried attempt reuses the same starting state), threaded through AdvanceSpec -> advance_dynamics -> back to the runner via new extract_dyn_state/restore_dyn_state helpers in traj_config.py. * Update task.py * Consolidate per-integrator dispatch in traj_config.py into a registry get_dynamics_cls, resolve_dyn_kws, prepare_atoms_for_dynamics, and the extract/restore_dyn_state pair each independently switched on which integrator (velocity-verlet/npt/mtknpt) was in play, via three different dispatch styles (if/elif on the config string, isinstance on the live ASE object). Replace all of it with one IntegratorSpec dataclass bundling a given integrator's ASE class, friendly-unit config schema, atom preprocessing, and state extract/restore, collected in a single INTEGRATORS registry. The five public functions become thin lookups against that registry; their signatures are unchanged so no call sites elsewhere need to change. * Update traj_config.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.