Adding support for Nicolet .e files

[MureziCapaul]

The reader for .e files as mentioned in #1510

[MureziCapaul]

Dear @zm711 , @h-mayorquin , @samuelgarcia
I contacted the developers of the Matlab modules some time ago. They responded quickly, saying neither of them had file specifications. They essentialy reverse-engineered the file using a hex-editor and some knowledge of the data structure. One of the maintainers provided me contacts of the developers of the company that developed the EEG-Systems. I contacted tow of them, but neither responded in over a month. I think it's fair to say that I will not be able to get hold of file specification.
So shall we continue with the pull request?
Since my last comment, we have been using the module in our center and I added some small fixes and improvements.
Also, I have almost 100 commits. Would it be possible to enable squashing for pull requests so the module becomes a single pull request? (https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/configuring-commit-squashing-for-pull-requests)

[h-mayorquin]

So shall we continue with the pull request?

Is this ready to be reviewed now?

Would it be possible to enable squashing for pull requests so the module becomes a single pull request

+1

[MureziCapaul]

@h-mayorquin , sorry for the delay. Yes, it is ready to be reviewed now.

[MureziCapaul]

@h-mayorquin Hi, I just wanted to ask about the current status of the pull request? Have you been able to finish code review? There are also some merge conflicts. Will you resolve them, or should I do it?

[h-mayorquin]

Hi @MureziCapaul, sorry for the long silence here, and thanks for the nudges. I got back this and I've stubbed the file to a CI-friendly size and will put it on the ephy_testing_data gin repo once I get another PR merged, so the tests here can run. I also pulled the MATLAB ieeg-portal/Nicolet-Reader you ported from (plus a second multi-segment .e) to check against.

I took a first pass and those are two comments:

1. Event/epoch times look like they come out scaled by the sampling rate. In _get_event_timestamps the timestamps and durations get multiplied by sampling_rates[seg_index][0] (so they're in samples), but _rescale_event_timestamp / _rescale_epoch_duration only .astype() without dividing back so the final times end up in samples rather than seconds (e.g. an event at 4 s in a 256 Hz file reports as 1024). The MATLAB reader keeps these in seconds (it uses etime(...), and stores duration as a float64 in seconds), so I think dropping the * sampling_rate is all it needs — that also fixes the t_start/t_stop filtering in the same method, which is currently comparing those sample values against second-valued bounds.

2. Small one in _parse_header: signal_buffers is built as np.array(["Signals", "0"], dtype=_signal_buffer_dtype). With the two-field dtype, numpy broadcasts each string across both fields and you get two records (('Signals','Signals') and ('0','0')) instead of one. A list-of-tuples fixes it: np.array([("Signals", "0")], dtype=_signal_buffer_dtype).

Neither is a big lift and the reader is clearly close. More to come, and thanks again for sticking with it.

[h-mayorquin]

Hi @MureziCapaul, before I commit the test fixture to the ephy_testing_data gin repo, I wanted you to have it so you can run the tests locally and confirm it reads correctly on your end.

One important thing: since .e files are human EEG (electroencephalography), and under the European data-protection rules (the General Data Protection Regulation, GDPR) EEG is treated as identifiable, special-category health data, I could not publish your recording as-is. So the fixture is a synthetic, de-identified version of it: the signal is per-channel time-shuffled (each channel's amplitude distribution is kept so the gains stay realistic, but it is no longer real brain data), and the record identifiers are neutralized (patientID and device_id zeroed, the annotator name blanked, all dates shifted back 20 years). The container structure, channel names, sampling rate, gains, events, and montage are untouched, so it exercises the reader exactly as before.

The attached zip has two files, and it would help if you could look at both:
nicolet_synthetic_for_review.zip

• test_synthetic.e is your file with only the signal reframed and the identifiers removed, structure otherwise intact. This is the one to diff against your original to confirm the anonymization did not lose anything important.
• test_synthetic_stub.e is the small (0.43 MB) version that will actually go to gin. It is the same file, additionally compacted (unread sections and unused padding removed) so it is small enough for continuous integration.

The SYNTHETIC_README.md in the zip lists every field that was changed.

Could you read both with the reader and confirm they come out correctly? Once you give the go-ahead I will commit test_synthetic_stub.e to gin as nicolet/e_files/test.e so the tests can run.

[MureziCapaul]

Hi @h-mayorquin Thank you very much for taking the time to review the code and de-identify the files. I can read both files with my reader without a problem, and as far as I can tell there is no difference between them and the original file (except for what you already said). I think you can commit the test_synthetic_stub.e to gin.

I also fixed the two issues you mentioned, and commited the fixes to my fork.

[h-mayorquin]

Hi @h-mayorquin Thank you very much for taking the time to review the code and de-identify the files. I can read both files with my reader without a problem, and as far as I can tell there is no difference between them and the original file (except for what you already said). I think you can commit the test_synthetic_stub.e to gin.

I also fixed the two issues you mentioned, and commited the fixes to my fork.

PR is up:
https://gin.g-node.org/NeuralEnsemble/ephy_testing_data/pulls/180

As long as someone from our team takes a look and is merge you will be able to develop against them. I added a mulit-segment data there.

[zm711]

I would prefer @apdavison and @JuliaSprenger to comment on this as regards the de-identification process as we recently had a meeting where (if I remember correctly) they had mentioned the rules for de-identification in Europe may be getting stricter. So before we merge the test file I would like them to comment.

[h-mayorquin]

I would prefer @apdavison and @JuliaSprenger to comment on this as regards the de-identification process as we recently had a meeting where (if I remember correctly) they had mentioned the rules for de-identification in Europe may be getting stricter. So before we merge the test file I would like them to comment.

I took that discussion into account and applied the principles that @JuliaSprenger mentioned there. That is, the data is fully synthetic and basically just imitates in form the container of the original files and fills both the arrays and attributes with noise and placeholders. So, same dress but empty synthethic content.