Skip to content

Support independent input and output steps - #860

Merged
alxmrs merged 4 commits into
mainfrom
feature/independent-prediction-horizon
Sep 4, 2026
Merged

Support independent input and output steps#860
alxmrs merged 4 commits into
mainfrom
feature/independent-prediction-horizon

Conversation

@alxmrs

@alxmrs alxmrs commented Aug 25, 2026

Copy link
Copy Markdown
Member

hist has been used to cover two concepts: the number of input steps and the number of output steps. These have always been decided together. This PR separates these two concepts so we can change them independently. Doing so, to me, clarifies what concept we actually need in various parts of how hist was used -- code paths either need input or output steps, and this PR makes use of the proper concept by name now instead of implicitly. I find that this makes reading various data flow paths more understandable than before.

🤖 below

Summary

  • add backwards-compatible data.output_steps configuration, defaulting to hist + 1
  • support rolling input histories when a model emits fewer timesteps than it consumes
  • anchor residual predictions to the most recent matching input span
  • separate input/output timestep handling in validation, rollout metrics, and Zarr writing
  • cover the 2-input / 1-output dataset, residual rollout, Trainer, validation, and writer paths

This enables hist: 1, output_steps: 1: two input states predict one future state, and each autoregressive call shifts [x(t-1), x(t)] to [x(t), x_hat(t+1)].

Existing configs omit output_steps and retain their current hist + 1 output blocks.

Validation

  • uvx pre-commit run --all-files
  • .venv/bin/pytest -m 'not manual and not cuda' -q
    • 417 passed, 2 skipped, 67 deselected, 10 xfailed

@alxmrs alxmrs left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initial review to agents.

self._tensor_map = tensor_map
self.num_prognostic_channels = num_prognostic_channels
self.hist = hist
self.input_hist = hist

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can make this input_steps that is hist+1 to be consistent with the accounting of output_steps, rather than use output_hist which is a weird concept.

Comment thread src/samudra/config.py Outdated
Comment thread src/samudra/models/base.py Outdated
Comment thread src/samudra/config.py Outdated
),
)
input_steps: int | None = Field(
default=None,

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be defaulted to 2 IMO, since his was defaulted to 1 before.

@alxmrs alxmrs changed the title Support independent input and output horizons Support independent input and output steps Aug 29, 2026
Comment thread src/samudra/config.py Outdated
@alxmrs
alxmrs requested a review from amogh-gulati August 31, 2026 17:11
@alxmrs

alxmrs commented Aug 31, 2026

Copy link
Copy Markdown
Member Author

@fomo-bot may I have your review? If you think the PR is ready for merge, will you approve it?

@fomo-bot fomo-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed current head 4ccafb7. The independent input/output step refactor looks ready to merge after focused local validation and green current-head CI.

@fomo-bot

Copy link
Copy Markdown
Collaborator

Approved PR #860.

I reviewed current head 4ccafb76aef3b613198e9c804f721597a58812bb, including live PR metadata, full diff, review threads, CI, and local validation. I did not find blocking issues. The remaining unresolved review threads are outdated, and their requested direction is reflected in the current code.

Validation:

  • git diff --check origin/main...HEAD passed.
  • Focused local suite passed: 249 passed, 65 deselected, 3 warnings.
  • The failed GPU workflow was an EC2 InsufficientInstanceCapacity launch failure; I reran failed jobs and the GPU run passed.
  • Final GitHub state: reviewDecision=APPROVED, mergeable=MERGEABLE, mergeStateStatus=CLEAN.
Debug info

@alxmrs

alxmrs commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

Hey @codex, will you review this PR?

@alxmrs
alxmrs force-pushed the feature/independent-prediction-horizon branch from 4ccafb7 to cfbf15c Compare September 1, 2026 22:41
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 4ccafb76ae

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@alxmrs

alxmrs commented Sep 2, 2026

Copy link
Copy Markdown
Member Author

@codex may I have your review on this PR?

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: cfbf15cfeb

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@alxmrs
alxmrs force-pushed the feature/independent-prediction-horizon branch from cfbf15c to 2a54ac2 Compare September 2, 2026 18:18
@alxmrs
alxmrs force-pushed the feature/independent-prediction-horizon branch from 662a87f to a66e9c6 Compare September 4, 2026 17:48

@amogh-gulati amogh-gulati left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! just one small comment which came up

Comment thread src/samudra/datasets.py Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🐑 : This is still old explanation right? where the inputs and outputs are coupled

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch—the docstring was still describing only the coupled legacy behavior. I updated it to define windows in terms of independent input_steps and output_steps, including explicit 2-in/2-out and 2-in/1-out examples (2f49ac9).

@alxmrs
alxmrs enabled auto-merge September 4, 2026 23:04
@alxmrs
alxmrs added this pull request to the merge queue Sep 4, 2026
Merged via the queue into main with commit 55fe01d Sep 4, 2026
20 checks passed
@alxmrs
alxmrs deleted the feature/independent-prediction-horizon branch September 4, 2026 23:35
@github-project-automation github-project-automation Bot moved this from Backlog to Done in Samudra Planning Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants