Skip to content

Add dataset-level recording duration and completeness fields to dataSummary - #12

Merged
neuromechanist merged 5 commits into
mainfrom
feature/data-summary-recording-duration
Aug 25, 2026
Merged

neuromechanist merged 5 commits into
mainfrom
feature/data-summary-recording-duration

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

  • Adds three optional, additive properties to the core dataSummary definition (schema/definitions/dataSummary.schema.json, $ref'd from schema/core/dataset.schema.json):
    • total_recording_duration: total recording duration in seconds, summed across recordings. Documents that concurrent channel groups within a single recording are not additive (a recording with simultaneous EEG and EOG streams contributes its longest stream, not the sum), and that the sum is a lower bound when recordings_unavailable is non-zero.
    • recording_duration_range: range of per-recording durations across the dataset, in seconds. Copies the exact {min, max} / additionalProperties: false shape of the existing channel_count_range and sampling_frequency_range.
    • recordings_unavailable: number of recordings that could not be summarised (truncated, corrupt, unsupported format, or otherwise unreadable). recording_count counts all recordings in the dataset, so recording_count - recordings_unavailable gives the number with usable summaries.
  • Bumps the version to 0.4.0 across all three pins: pyproject.toml:7, src/neuroschema/__init__.py:3, README.md:77. This is a minor bump, not a patch bump: dataSummary.schema.json is $ref'd directly from schema/core/dataset.schema.json, so these are new core fields, which .rules/schema-design.md places at MINOR ("New core fields, new extension namespaces"). This also matches project precedent -- v0.2.0 and v0.3.0 were both minor bumps for new core dataset-level fields.

Why

This unblocks a downstream NEMAR dataset-duration search facet: NEMAR wants to let users search datasets by total signal duration, but dataSummary is additionalProperties: false with no dataset-level duration field (signalSummary.recording_duration is per-record only), so a consumer cannot emit the field until the schema declares it.

recordings_unavailable closes a related completeness gap: a summary generated from a serving catalog (a viewer index, a converted-copy manifest) only sees what converted successfully. Without a field for the shortfall, an unreadable recording silently vanishes from recording_count and the dataset looks smaller than it actually is, with nothing to indicate anything is missing. This is a general FAIR-completeness concern, not NEMAR-specific: a consumer should be able to tell "this dataset has 124 recordings" from "this dataset has 126 recordings, 2 of which we could not read."

Both changes are additive and backward compatible: no existing field changes, and nothing becomes required.

Test plan

  • Added TestV040CoreFields in tests/test_schema_validation.py, following the conventions of the existing TestV030CoreFields section.
  • total_recording_duration: accepts a positive number, accepts null, rejects a negative value.
  • recording_duration_range: accepts {min, max}, accepts null, rejects an unknown key (proves additionalProperties: false is enforced).
  • recordings_unavailable: accepts a non-negative integer, accepts null, rejects a negative value.
  • A dataSummary with none of the new fields still validates (backward compatibility).
  • Each rejection test was verified against a known-broken state before being trusted: temporarily removing minimum: 0 or flipping additionalProperties to true made the corresponding test fail, then the schema was reverted and git status --porcelain/git diff confirmed only the intended change remained.
  • pytest: 71 passed.
  • ruff check .: all checks passed. ruff format --check .: all files already formatted.

Adds two optional, additive properties to the core dataSummary
definition so consumers (NEMAR) can surface a dataset-level recording
duration facet. total_recording_duration sums per-recording duration
in seconds, noting that concurrent channel groups within one recording
are not additive. recording_duration_range mirrors the existing
channel_count_range/sampling_frequency_range shape.
Covers total_recording_duration (number/null, rejects negative) and
recording_duration_range (min/max object, null, rejects unknown key),
plus a backward-compatibility check for data_summary with neither
field set.
…ound

recordings_unavailable counts recordings that could not be summarised
(truncated, corrupt, unsupported format, unreadable), closing a
completeness gap where index-derived recording_count silently
undercounts unreadable recordings. total_recording_duration's
description now notes it is a lower bound when recordings_unavailable
is non-zero, since the sum only covers recordings that could be read.
Covers recordings_unavailable (integer/null, rejects negative) and
renames the v0.3.1 test section to v0.4.0 to match the corrected
version bump (new core fields require a minor bump per
.rules/schema-design.md, not a patch bump).
New core fields on dataSummary (total_recording_duration,
recording_duration_range, recordings_unavailable) require a minor
bump per .rules/schema-design.md, not the patch bump originally
proposed.
@neuromechanist
neuromechanist merged commit d920ad0 into main Aug 25, 2026
5 checks passed
@neuromechanist
neuromechanist deleted the feature/data-summary-recording-duration branch August 25, 2026 05:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant