fix: fall back to frame-alignment check when meta lacks fileSizeBytes - #4
Merged
Merged
Conversation
SpikeGLX writes fileSizeBytes, fileTimeSecs, fileSHA1, and the hardware error-flag summary only in the final step of its shutdown sequence, after acquisition data is fully flushed to disk. If the app is closed or crashes during that finalization step (e.g. while still computing the SHA1 checksum of a large .bin file), the resulting .meta file is missing these trailing fields even though the recorded data itself is complete. This previously caused a hard KeyError in validate_file, crashing EphysRecording ingestion for otherwise-usable recordings (confirmed on two live cases: the .ap.bin files were exact multiples of the per-sample-frame size, with no evidence of truncation). When fileSizeBytes is absent, fall back to checking that the file size divides evenly into whole sample frames (nSavedChans * 2 bytes) instead of failing outright. A genuinely truncated/corrupted file will not be frame-aligned and will still raise. Logs a warning so these recordings remain visible for follow-up. Assisted-by: ClaudeCode:claude-sonnet-5
This was referenced Aug 17, 2026
tabedzki
added a commit
that referenced
this pull request
Aug 17, 2026
Reverts the frame-alignment fallback added in #4. The root cause (SpikeGLX crashing before finishing its shutdown sequence, so fileSizeBytes/fileTimeSecs/fileSHA1 were never written) has been fixed upstream in SpikeGLX itself. These fields are relied on downstream for values we need, so validate_file should go back to treating their absence as an error rather than silently tolerating it. Previously-affected files will be corrected by a separate one-off script. This reverts commit 0f2e909. Assisted-by: ClaudeCode:claude-sonnet-5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 Human guided, AI assisted PR (using this skill). AI text below. 🤖
Summary
Automated ingestion failed with:
raised from
SpikeGLX.validate_file, on two separate recordings from the same rig/mouse (emdia_nicky, 2026-03-30 and 2026-04-10, ~2 weeks apart), with normal successful recordings from the same setup on either side of both dates.Investigation
Comparing the failing
.ap.metaagainst a known-good one from the same probe/mouse (diff <(sort ...)) showed the failing file is missing exactly four fields relative to the good one:Per SpikeGLX's metadata behavior, these fields are computed and written only as the last step of its shutdown sequence — after acquisition data is fully flushed to disk, and after computing a SHA1 checksum of the entire
.ap.binfile (which can take real wall-clock time on a large file). Everything that comes before these fields in the.metafile (channel maps, IMRO table, geometry map, etc.) was present and well-formed in both failing cases.Directly checked the
.ap.binfile size against the expected per-sample-frame size (nSavedChans * 2bytes for int16 data): it divided evenly with no remainder, and the implied duration (~57 min) was a plausible full recording — i.e. the actual recorded data is complete and not truncated. This is consistent with the SpikeGLX app being closed, crashed, or killed sometime after finishing the data write but before finishing its own finalization/checksum step, rather than any data corruption or transfer issue.This is happening more than once on the same rig, so a hard crash in ingestion for every occurrence is worth avoiding — the data itself is fine.
Change
SpikeGLX.validate_file: whenfileSizeBytesis absent from the meta file, fall back to checking that the actual.binfile size divides evenly into whole sample frames instead of raising immediately. A genuinely truncated/corrupted file (mid-frame cutoff) will still fail this check and still raiseIOError. Logs a warning when this fallback path is used, so these recordings stay visible for follow-up rather than silently passing.Test plan
20260330_g0,20260410_g0) and confirmEphysRecordingpopulates successfully with a duration computed from the actual.binfile.metafile (withfileSizeBytespresent) still validates via the original exact-match path.binfile (not frame-aligned) still raisesIOErrorAssisted-by: ClaudeCode:claude-sonnet-5