Repository navigation
Add dataset-level recording duration and completeness fields to dataSummary - #12
Merged
Merged
Conversation
Adds two optional, additive properties to the core dataSummary definition so consumers (NEMAR) can surface a dataset-level recording duration facet. total_recording_duration sums per-recording duration in seconds, noting that concurrent channel groups within one recording are not additive. recording_duration_range mirrors the existing channel_count_range/sampling_frequency_range shape.
Covers total_recording_duration (number/null, rejects negative) and recording_duration_range (min/max object, null, rejects unknown key), plus a backward-compatibility check for data_summary with neither field set.
…ound recordings_unavailable counts recordings that could not be summarised (truncated, corrupt, unsupported format, unreadable), closing a completeness gap where index-derived recording_count silently undercounts unreadable recordings. total_recording_duration's description now notes it is a lower bound when recordings_unavailable is non-zero, since the sum only covers recordings that could be read.
Covers recordings_unavailable (integer/null, rejects negative) and renames the v0.3.1 test section to v0.4.0 to match the corrected version bump (new core fields require a minor bump per .rules/schema-design.md, not a patch bump).
New core fields on dataSummary (total_recording_duration, recording_duration_range, recordings_unavailable) require a minor bump per .rules/schema-design.md, not the patch bump originally proposed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
dataSummarydefinition (schema/definitions/dataSummary.schema.json,$ref'd fromschema/core/dataset.schema.json):total_recording_duration: total recording duration in seconds, summed across recordings. Documents that concurrent channel groups within a single recording are not additive (a recording with simultaneous EEG and EOG streams contributes its longest stream, not the sum), and that the sum is a lower bound whenrecordings_unavailableis non-zero.recording_duration_range: range of per-recording durations across the dataset, in seconds. Copies the exact{min, max}/additionalProperties: falseshape of the existingchannel_count_rangeandsampling_frequency_range.recordings_unavailable: number of recordings that could not be summarised (truncated, corrupt, unsupported format, or otherwise unreadable).recording_countcounts all recordings in the dataset, sorecording_count - recordings_unavailablegives the number with usable summaries.pyproject.toml:7,src/neuroschema/__init__.py:3,README.md:77. This is a minor bump, not a patch bump:dataSummary.schema.jsonis$ref'd directly fromschema/core/dataset.schema.json, so these are new core fields, which.rules/schema-design.mdplaces at MINOR ("New core fields, new extension namespaces"). This also matches project precedent -- v0.2.0 and v0.3.0 were both minor bumps for new core dataset-level fields.Why
This unblocks a downstream NEMAR dataset-duration search facet: NEMAR wants to let users search datasets by total signal duration, but
dataSummaryisadditionalProperties: falsewith no dataset-level duration field (signalSummary.recording_durationis per-record only), so a consumer cannot emit the field until the schema declares it.recordings_unavailablecloses a related completeness gap: a summary generated from a serving catalog (a viewer index, a converted-copy manifest) only sees what converted successfully. Without a field for the shortfall, an unreadable recording silently vanishes fromrecording_countand the dataset looks smaller than it actually is, with nothing to indicate anything is missing. This is a general FAIR-completeness concern, not NEMAR-specific: a consumer should be able to tell "this dataset has 124 recordings" from "this dataset has 126 recordings, 2 of which we could not read."Both changes are additive and backward compatible: no existing field changes, and nothing becomes required.
Test plan
TestV040CoreFieldsintests/test_schema_validation.py, following the conventions of the existingTestV030CoreFieldssection.total_recording_duration: accepts a positive number, acceptsnull, rejects a negative value.recording_duration_range: accepts{min, max}, acceptsnull, rejects an unknown key (provesadditionalProperties: falseis enforced).recordings_unavailable: accepts a non-negative integer, acceptsnull, rejects a negative value.dataSummarywith none of the new fields still validates (backward compatibility).minimum: 0or flippingadditionalPropertiestotruemade the corresponding test fail, then the schema was reverted andgit status --porcelain/git diffconfirmed only the intended change remained.pytest: 71 passed.ruff check .: all checks passed.ruff format --check .: all files already formatted.