☕️ Chicago's second favorite bean
This repo provides a configurable way to collate data from multiple sources into a single denormalized dataframe and create tokenized timelines from the results. It benefits from previous experience collating data to train foundation models on tokenized electronic health records. 1 2 3 4 5
Install the latest release from PyPI:
pip install cocoa-tokenizerThis installs the cocoa command. To work from source instead (e.g. for
development):
git clone git@github.com:bbj-lab/cocoa.git
cd cocoa
python -m venv .venv
. .venv/bin/activate
pip install -e .The collator pulls from raw data tables (parquet or csv) and combines them into a
single denormalized dataframe in a
MEDS-like format. Each row
in the output represents an event with a subject_id, time, code (all
mandatory), and optional numeric_value / text_value columns.
Collation is driven by a YAML config (the package ships a default; see
./src/cocoa/config/collation.yaml)
that specifies:
- A reference table with a primary key (
subject_id), start/end times, and optional augmentation joins (e.g. joining a patient demographics table). - A list of entries, each mapping a source table (or the reference frame
itself via
table: REFERENCE) to the output schema. Each entry declares which column provides thecode,time, and optionallynumeric_value, andtext_value. Codes can be given a prefixprefix. Some preprocessing can be done with optional entries forfilter_expr,with_col_expr,agg_expr, andkey. These take the form of polars expressions that are evaluated and applied to the dataframe during loading. Mild checks are performed when evaluating these expressions, but in general, the yaml config is just as powerful as the python. Check all yaml files prior to use. - Subject splits (
train_frac/tuning_frac) that partition subjects chronologically into train, tuning, and held-out sets.
A collation config has three top-level sections: identifiers, subject splits, and the reference + entries that define which events to extract.
subject_id: hospitalization_id # the atomic unit of interest
group_id: patient_id # multiple subjects can belong to a group
default_timezone: UTC # or, e.g. America/Chicago; times are stored in this zone
default_time_unit: us # precision of stored times: 'ms', 'us', or 'ns'
drop_duplicate_events: false # collapse rows an entry repeats verbatim
subject_splits:
train_frac: 0.7
tuning_frac: 0.1
# the remainder is held outsubject_id is the column that uniquely identifies each subject (e.g. a
hospitalization). group_id is an optional higher-level grouping column.
Subjects are sorted chronologically and split into train / tuning / held-out sets
according to the specified fractions.
All timestamps are stored as timezone-aware datetimes in default_timezone (an
IANA name like America/Chicago, defaulting to UTC if unset). Columns that
already carry a timezone are converted instant-preserving; columns without one
are assumed to be local times in default_timezone and labeled as such. Where a
daylight-saving shift makes such a local time ambiguous, the later instant is
used; a local time that a shift skips over is an error. A csv carries no schema,
so its datetimes arrive as strings and are parsed under the same two rules: one
written with a UTC offset is converted, one without is localized.
The timezone is part of the stored dtype, so it travels with the data through the
rest of the pipeline. Durations are computed from instants and are therefore
unaffected by the choice: time spacing tokens, and the winnowing thresholds and
horizons, measure elapsed physical time across a daylight-saving shift. Clock
tokens are different — CLCK//HH marks an hour of the local day in
default_timezone (see Tokenization). And because processed
timestamps keep their timezone, cocoa combine-datasets cannot merge processed
directories that were collated with different default_timezone values.
Timestamps are stored at default_time_unit precision, one of the three polars
datetime units — ms, us (the default), or ns. Raw columns are repinned to
it on load whatever precision they arrive at, so a single unit holds across
tables; set it to match an upstream table or a downstream consumer. As with the
timezone, the unit is part of the stored dtype and travels with the data, so
combine-datasets likewise cannot merge directories collated at different units.
Durations are derived from the stored instants, so the choice changes no token or
label — but note that ns cannot represent times outside 1677–2262.
drop_duplicate_events collapses duplicate events as each entry is collated. Raw
tables can carry the same event more than once — a result posted by two systems,
a vital charted twice, a row multiplied by a join — and each copy would otherwise
become another token at the same timestamp. Rows are compared on the whole
collated event, so two copies are dropped to one only when their subject_id,
time, code, numeric_value, and text_value all agree.
The reference table is the primary static table to which other static information can be joined:
reference:
table: clif_hospitalization
start_time: admission_dttm
end_time: discharge_dttm
augmentation_tables:
- table: clif_patient
key: patient_id
validation: "m:1"
with_col_expr: pl.lit("AGE").alias("AGE")table— the name of the parquet (or csv) file in--raw-data-home(without the extension).start_time/end_time— columns that define the subject's time window; used to filter events from other tables whenreference_keyis set (see below).augmentation_tables— optional list of tables to join onto the reference frame. Each needs akeyto join on and avalidationmode (e.g."m:1"). You can also add computed columns viawith_col_expr.
The pass_through_columns option allows you to preserve static columns from the
reference table and include them in the output files. This is useful for
demographic and contextual data that should accompany the collated events:
pass_through_columns:
- age_at_admission
- admission_type_category
...Columns specified in this list will be copied from the reference table to:
subject_splits.parquet*_for_inference.parquetfiles (e.g.,train_for_inference.parquet,tuning_for_inference.parquet,held_out_for_inference.parquet) — for use in downstream tasks where you may need subject metadata alongside predictions
The entries list defines the events to extract. Every entry produces rows with
the columns subject_id, time, code, numeric_value, and text_value. The
entry's fields tell the collator which source columns map to these outputs.
Required fields:
| Field | Description |
|---|---|
table |
Source table name, or REFERENCE to pull from the reference frame. |
code |
Column whose values become the event code. |
time |
Column whose values become the event timestamp. |
Optional fields:
| Field | Description |
|---|---|
prefix |
String prepended to the code (separated by //), e.g. LAB-RES. |
numeric_value |
Column to use as the numeric value for the event. |
text_value |
Column to use as the text value for the event. |
filter_expr |
A Polars expression (or list of expressions) to filter rows before extraction. |
with_col_expr |
A Polars expression (or list) to add computed columns before extraction. |
agg_expr |
A Polars aggregation expression (or list) applied via group_by(...).agg(...) before extraction. |
key |
Grouping key used with agg_expr. Defaults to subject_id when not provided. |
reference_key |
Join the source table to the reference frame on this key and keep only rows within the subject's start_time–end_time window. |
Examples:
-
A simple categorical event from the reference frame:
- table: REFERENCE prefix: DSCG code: discharge_category time: discharge_dttm
creates codes such as
DSCG//assisted_living,DSCG//home,DSCG//hospicewith timedischarge_dttm. -
A numeric event from an external table:
- table: clif_labs prefix: LAB-RES code: lab_category numeric_value: lab_value_numeric time: lab_result_dttm
creates codes such as
LAB-RES//altandLAB-RES//astwith numeric_valuelab_value_numericat timelab_result_dttm. -
Tables can be filtered prior to extraction with
filter_expr:- table: clif_position prefix: POSN filter_expr: pl.col("position_category") == "prone" code: position_category time: recorded_dttm
selects only rows where
pl.col("position_category") == "prone" -
Multiple filters can be applied as a list:
- table: clif_medication_admin_intermittent_converted prefix: MED-INT filter_expr: - pl.col("mar_action_category") == "given" - pl.col("_convert_status") == "success" code: med_category numeric_value: med_dose_converted time: admin_dttm
-
Pre-aggregating events before token extraction with
agg_expr:- table: clif_crrt_therapy prefix: LABEL filter_expr: pl.col("crrt_mode_category").is_not_null() with_col_expr: pl.lit("crrt_init").alias("code") agg_expr: - pl.col("code").first() - pl.col("recorded_dttm").first() code: code time: recorded_dttm
-
Creating a computed column with
with_col_exprto use as the code:- table: clif_respiratory_support_processed prefix: RESP with_col_expr: pl.lit("fio2_set").alias("code") filter_expr: pl.col("fio2_set").is_finite() code: code numeric_value: fio2_set time: recorded_dttm
-
The
reference_keycan be used to restrict events to a subject's time window:- table: clif_code_status prefix: CODE code: code_status_category time: admission_dttm reference_key: patient_id
-
meds.parquetgives a table of the collated events:┌────────────┬─────────────────────┬──────────────────────────────┬───────────────┬────────────┐ │ subject_id ┆ time ┆ code ┆ numeric_value ┆ text_value │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ datetime[μs] ┆ str ┆ f32 ┆ str │ ╞════════════╪═════════════════════╪══════════════════════════════╪═══════════════╪════════════╡ │ 24591817 ┆ 2111-09-26 18:15:00 ┆ MED-CTS//sodium_chloride ┆ 0.0 ┆ null │ │ 21343412 ┆ 2112-01-11 06:31:00 ┆ LAB-RES//albumin ┆ 3.3 ┆ null │ │ 24894995 ┆ 2113-01-14 14:25:00 ┆ LAB-ORD//creatinine ┆ null ┆ null │ │ 20947416 ┆ 2110-12-12 18:41:00 ┆ LAB-RES//hemoglobin ┆ 8.4 ┆ null │ │ 25082363 ┆ 2110-06-17 17:00:00 ┆ VTL//respiratory_rate ┆ 30.0 ┆ null │ │ … ┆ … ┆ … ┆ … ┆ … │ │ 22074503 ┆ 2110-07-13 03:53:00 ┆ LAB-ORD//chloride ┆ null ┆ null │ │ 24524153 ┆ 2110-10-08 03:20:00 ┆ LAB-RES//glucose_serum ┆ 179.0 ┆ null │ │ 28104308 ┆ 2112-03-22 14:31:00 ┆ LAB-RES//sodium ┆ 137.0 ┆ null │ │ 23859742 ┆ 2110-08-21 21:35:00 ┆ LAB-RES//ptt ┆ 26.299999 ┆ null │ │ 25805890 ┆ 2110-10-03 11:00:00 ┆ LAB-ORD//eosinophils_percent ┆ null ┆ null │ └────────────┴─────────────────────┴──────────────────────────────┴───────────────┴────────────┘ -
subject_splits.parquetgives a table of all subject_id's and their corresponding split assignment:┌────────────┬──────────┐ │ subject_id ┆ split │ │ --- ┆ --- │ │ str ┆ str │ ╞════════════╪══════════╡ │ 21081215 ┆ train │ │ 20302177 ┆ train │ │ … ┆ … │ │ 27116134 ┆ tuning │ │ 29134959 ┆ tuning │ │ … ┆ … │ │ 28150003 ┆ held_out │ │ 22151813 ┆ held_out │ └────────────┴──────────┘
The tokenizer consumes the collated parquet output and converts events into integer token sequences suitable for sequence models. It:
- Adds
BOS/EOS(beginning/end-of-sequence) tokens to each subject's timeline. - Optionally inserts configurable clock tokens to mark the passage of time.
- Computes quantile-based bins for numeric values (from training data only).
- Optionally inserts configurable time spacing tokens between events.
- Maps codes (and optionally their binned values) to integer tokens via a vocabulary that is formed during training and is frozen for tuning/held-out data.
- Aggregates per-subject token sequences according to time, and then configurable sort order.
Tokenization is driven by its own YAML config (the package ships a default; see
./src/cocoa/config/tokenization.yaml)
that specifies:
n_bins— number of quantile bins for numeric values.fused— whether to fuse the code, binned value, and text value into a single token (true, the default) or keep them as separate tokens (false).include_numeric_values— whether to include raw numeric values alongside tokens in the output (falseby default).include_hours_to_end_time— whether to include, alongside tokens, the fractional hours from each token to its subject'send_timeas recorded insubject_splits.parquet(falseby default). Values count down toward the end time and go negative for tokens beyond it, which theEOSand time spacing tokens of a timeline running pastend_timecan be.min_training_ct— minimum number of times a word must occur in the training split to earn a place in the vocabulary.0(the default) disables the threshold and keeps every word seen while training. Raise it to prune the long tail of one-off codes. Structural tokens (BOS,EOS, and anyCLCK//orTIME//tokens) are exempt and always learned.insert_spacers— whether to insert time spacing tokens between events.insert_clocks— whether to insert clock tokens at specified times.ordering— the priority order of code prefixes when sorting events within the same timestamp. A prefix missing from it sorts after the listed ones, though still beforeEOS, which always ends the timeline.spacers— mapping of time intervals (e.g.,5m-15m,1h-2h) to their lower bounds in minutes, used for time spacing tokens.clocks— list of hour strings (e.g.,00,04, ...) at which to insert clock tokens. Hours are read in the collation config'sdefault_timezone, so across a daylight-saving shift a repeated hour yields two tokens and a skipped hour yields none.
-
tokens_times.parquetgives one row per subject. By default it has three columns:subject_idtokens— the integer token sequence for the subject's timeline.times— a parallel list of timestamps, one per token, indicating when each event occurred.
A
numeric_valuescolumn, holding the corresponding values for numeric value tokens, is added only wheninclude_numeric_valuesis set, and anhours_to_end_timecolumn only wheninclude_hours_to_end_timeis.The table will look something like this:
┌────────────────────┬─────────────────┬─────────────────────────────────┐ │ subject_id ┆ tokens ┆ times │ │ --- ┆ --- ┆ --- │ │ str ┆ list[u32] ┆ list[datetime[μs]] │ ╞════════════════════╪═════════════════╪═════════════════════════════════╡ │ 20002103 ┆ [20, 350, … 21] ┆ [2116-05-08 02:45:00, 2116-05-… │ │ 20008372 ┆ [20, 350, … 21] ┆ [2110-10-30 13:03:00, 2110-10-… │ │ … ┆ … ┆ … │ │ 29994865 ┆ [20, 364, … 21] ┆ [2111-01-28 21:49:00, 2111-01-… │ └────────────────────┴─────────────────┴─────────────────────────────────┘In this example, token 20 corresponds to the beginning-of-sequence token (
BOS), token 21 to the end-of-sequence token (EOS), and the tokens in between correspond to the subject's clinical events in chronological order (with ties broken by the configuredordering). In fused mode each event is a single token; in unfused mode an event with a numeric value becomes two tokens (code + quantile bin). -
tokenizer.yamlis a plain yaml file that contains information about the configuration, learned vocabulary, and bins. This file is sufficient to reconstitute the tokenizer object. Currently, there's an entry for the lookup that maps strings to tokens:lookup: UNK: 0 ADMN//direct: 1 ADMN//ed: 2 ADMN//elective: 3 AGE//age_Q0: 4 …
and an entry for bin cutpoints:
bins: VTL//heart_rate: - 65.0 - 70.0 - 75.0 - 80.0 - 84.0 - 89.0 - 94.0 - 100.0 - 108.0 LAB-RES//platelet_count: - 62.0 - 114.0 - 147.0 - 175.0 - 203.0 - 233.0 - 267.0 - 314.0 - 390.0 …
The lists following each key correspond to the cutpoints for the associated category.
Tip
To train a generative event model on this data, check out our configurable trainer: 🦜 cotorra
The winnower prepares held-out timelines for evaluation by filtering and flagging subjects based on outcome criteria. It:
- Loads held-out data from the tokenized timelines and associated timestamps.
- Splits each subject's timeline at a configurable time horizon or at the first occurrence of a specified token, separating events into "past" (before the horizon) and "future" (after the horizon).
- Checks for the presence of outcome tokens in both the past and future periods.
- Filters out subjects whose timelines don't exceed the horizon duration, ensuring subjects have sufficient observation time.
- Outputs a winnowed dataset suitable for inference and evaluation tasks.
Winnowing is driven by a YAML config (the package ships a default; see
./src/cocoa/config/winnowing.yaml)
that specifies:
outcome_tokens— list of event codes to track as outcomes (e.g.,XFR-IN//icu,DSCG//expired). The winnower creates binary flags for each outcome indicating whether that token appears in the past or future period.threshold— defines how the threshold is set. Currently supported options are as follows:duration_s(integer) thresholds after a given duration (in seconds)first_occurrence(token string) thresholds after the first occurrence of the provided token
horizon_after_threshold_sis an optional parameter that allows you to set a prediction window (in seconds) after the threshold is triggeredsplits— an optional list selecting which splits to prepare (defaults toheld_out). Rep-based inference also requires thetrainandtuningsplits.
Example configuration:
outcome_tokens: # supports patterns with fnmatch
- XFR-IN//icu # ICU transfer
- RESP//imv # invasive mechanical ventilation event
- DSCG//expired # discharge due to death
- LABEL//* # any kind of label token
threshold:
# choose one and only one of the following
duration_s: !!int 86400 # 24h
# first_occurrence: XFR-IN//icu
horizon_after_threshold_s: !!int 2592000 # 30d outcome window after prediction threshold
splits: # select which splits to prepare
- train
- tuning
- held_outheld_out_for_inference.parquethas columns for each outcome token (e.g.,XFR-IN//icu_past,XFR-IN//icu_future) indicating whether that outcome occurred in the respective time period.train_for_inference.parquetandtuning_for_inference.parquetare also provided; these are required to make rep-based predictions
These files are written to the processed data directory unless cocoa winnow is
given --output-home / -o, which sends them to another directory (created if
need be) and leaves the processed data directory untouched. The files of the
processed data directory are copied there too, so that it is a complete processed
dataset of its own (one cocoa visualize -p can read, say); its subdirectories
are not, nor are any *_for_inference.parquet files already there, which may
come from another configuration. That way one tokenized dataset can be winnowed
under several configurations, each into its own directory:
cocoa winnow -p ./processed/mimic -o ./processed/mimic/winnowed-24h
cocoa winnow -p ./processed/mimic -o ./processed/mimic/winnowed-icu \
'~threshold.duration_s' threshold.first_occurrence=XFR-IN//icuWe provide a CLI that should be sufficient for most use cases (to change a few config keys from the command line, see Overriding config keys):
Usage: cocoa [OPTIONS] COMMAND [ARGS]...
Configurable collation and tokenization (v26.6.1)
╭─ Options ───────────────────────────────────────────────────────────────╮
│ --install-completion Install completion for the current │
│ shell. │
│ --show-completion Show completion for the current shell, │
│ to copy it or customize the │
│ installation. │
│ --help -h Show this message and exit. │
╰─────────────────────────────────────────────────────────────────────────╯
╭─ Commands ──────────────────────────────────────────────────────────────╮
│ collate Collate raw data into a denormalized format. │
│ tokenize Tokenize collated data into integer sequences. │
│ winnow Winnow held-out data for evaluation. │
│ pipeline Run the full pipeline: collate, tokenize, & winnow. │
│ combine-datasets Combine multiple processed datasets into one. │
╰─────────────────────────────────────────────────────────────────────────╯
with commands:
-
cocoa collateUsage: cocoa collate [OPTIONS] [overrides]... Collate raw data into a denormalized format. Reads collation configuration and produces a MEDS-like parquet file with collated events. ╭─ Arguments ─────────────────────────────────────────────────────────────╮ │ overrides <str> Config overrides: key=value sets a key, adding │ │ it if need be, and ~key deletes one │ ╰─────────────────────────────────────────────────────────────────────────╯ ╭─ Options ───────────────────────────────────────────────────────────────╮ │ --collation-config -c <path> Collation configuration file │ │ (overrides default) │ │ * --raw-data-home -r <str> Raw data directory [required] │ │ * --processed-data-home -p <str> Processed data directory │ │ [required] │ │ --verbose -v Verbose logging for collate; │ │ this may cause memory issues │ │ with large datasets │ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────╯ -
cocoa tokenizeUsage: cocoa tokenize [OPTIONS] [overrides]... Tokenize collated data into integer sequences. Reads collated parquet files and produces tokenized timelines with vocabulary and bin information. ╭─ Arguments ─────────────────────────────────────────────────────────────╮ │ overrides <str> Config overrides: key=value sets a key, adding │ │ it if need be, and ~key deletes one │ ╰─────────────────────────────────────────────────────────────────────────╯ ╭─ Options ───────────────────────────────────────────────────────────────╮ │ --tokenization-config -c <path> Tokenization configuration │ │ file (overrides default) │ │ * --processed-data-home -p <str> Processed data directory │ │ [required] │ │ --tokenizer-home -t <str> Load a previously learned │ │ tokenizer from this │ │ tokenizer.yaml file (reuses │ │ its frozen vocabulary and │ │ bins) │ │ --verbose -v Verbose logging for tokenize; │ │ this may cause memory issues │ │ with large datasets │ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────╯ -
cocoa winnowUsage: cocoa winnow [OPTIONS] [overrides]... Winnow held-out data for evaluation. Filters held-out timelines and assigns flags to disqualify certain subjects from evaluation based on the configured criteria. ╭─ Arguments ─────────────────────────────────────────────────────────────╮ │ overrides <str> Config overrides: key=value sets a key, adding │ │ it if need be, and ~key deletes one │ ╰─────────────────────────────────────────────────────────────────────────╯ ╭─ Options ───────────────────────────────────────────────────────────────╮ │ --winnowing-config -c <path> Winnowing configuration file │ │ (overrides default) │ │ * --processed-data-home -p <str> Processed data directory │ │ [required] │ │ --output-home -o <str> Directory to write the │ │ winnowed files to, along with │ │ copies of the processed data │ │ directory's files (defaults │ │ to the processed data │ │ directory) │ │ --verbose -v Verbose logging for winnow; │ │ prints summary statistics │ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────╯ -
cocoa pipelineUsage: cocoa pipeline [OPTIONS] [overrides]... Run the full pipeline: collate, tokenize, & winnow. ╭─ Arguments ─────────────────────────────────────────────────────────────╮ │ overrides <str> Config overrides, each key starting with the │ │ stage it's for: e.g. tokenization.n_bins=5 or │ │ ~winnowing.threshold.duration_s │ ╰─────────────────────────────────────────────────────────────────────────╯ ╭─ Options ───────────────────────────────────────────────────────────────╮ │ --collation-config <path> Collation configuration file │ │ (overrides default) │ │ --tokenization-config <path> Tokenization configuration │ │ file (overrides default) │ │ --winnowing-config <path> Winnowing configuration file │ │ (overrides default) │ │ * --raw-data-home -r <str> Raw data directory [required] │ │ * --processed-data-home -p <str> Processed data directory │ │ [required] │ │ --verbose -v Verbose logging for pipeline │ │ steps │ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────╯ -
cocoa visualizeUsage: cocoa visualize [OPTIONS] {subject_id} [overrides]... Visualize a subject's timeline as an interactive html page. Serves the page on localhost until interrupted, or saves it as a self-contained html file with --export-html, a static pdf with --export-pdf, or both. ╭─ Arguments ─────────────────────────────────────────────────────────────╮ │ * subject_id <str> Subject whose timeline to show [required] │ │ overrides <str> Config overrides: key=value sets a key, │ │ adding it if need be, and ~key deletes one │ ╰─────────────────────────────────────────────────────────────────────────╯ ╭─ Options ───────────────────────────────────────────────────────────────╮ │ * --processed-data-home -p <str> Processed data directory │ │ [required] │ │ --visualization-config -c <path> Visualization configuration │ │ file (overrides default) │ │ --export-html -e <path> Save the timeline to this │ │ self-contained html file │ │ instead of serving it │ │ --export-pdf <path> Save a static pdf of the │ │ timeline to this file │ │ instead of serving it; with │ │ --export-html, save both │ │ --host <str> Address to serve the page on │ │ [default: 127.0.0.1] │ │ --port <int> Port to serve the page on; 0 │ │ picks any │ │ [default: 8765] │ │ --open Open the page in a web │ │ browser │ │ --show-winnowing Show the winnowing split │ │ into past and future, and │ │ its outcome flags │ │ --help -h Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────╯
To change a few keys without writing a new config file, list them after a command's options, as dotted paths into its config:
cocoa tokenize -p processed/mimic n_bins=5 insert_spacers=truecocoa pipeline takes them too, each key starting with the stage it's for:
cocoa pipeline -r raw/mimic -p processed/mimic \
collation.default_timezone=America/Chicago tokenization.n_bins=5 \
'~winnowing.threshold.duration_s' winnowing.threshold.first_occurrence=XFR-IN//icuIt checks every stage's overrides before collating, so an unreadable winnowing override fails at once rather than after the first two stages have run.
The syntax borrows from Hydra's:
key=valuesets a key, adding it if it isn't there, e.g. ahorizon_after_threshold_sfor winnowing. Keys aren't checked against the config, so a misspelled one is added quietly and does nothing. A leading+or++, as Hydra marks an addition, is accepted and ignored.~keydeletes a key.
Overrides apply in order to the config file passed with -c (or to the shipped
default when none is). Values are read as YAML: 0.5 is a float, [a, b] a
list, and {train_frac: 0.5} a block, merged into any block already there.
A few things to know:
- Setting a key to
nulldoesn't switch it off where its presence is what counts. The winnower uses whichever ofthreshold.duration_sandthreshold.first_occurrenceis there, so switching between them takes~on one and setting the other, as above. - Lists are replaced whole:
entriescan't be edited one item at a time, so a change to an entry calls for a config file. - YAML decides each value's type, so quote what has to stay a string:
'clocks=["00", "12"]', since an unquoted00is the number 0, and'other_color="#737373"', since an unquoted#starts a comment. - With
--tokenizer-home, the loaded tokenizer's saved config wins over overrides, as it does over-c. - Quote anything starting with
~in zsh, which otherwise reads it as a directory name and fails.
Tip
For common use cases, check out the recipes section!
Footnotes
-
M. Burkhart, B. Ramadan, Z. Liao, K. Chhikara, J. Rojas, W. Parker, & B. Beaulieu-Jones, Foundation models for electronic health records: representation dynamics and transferability, arXiv:2504.10422 ↩
-
M. Burkhart, B. Ramadan, L. Solo, W. Parker, & B. Beaulieu-Jones, Quantifying surprise in clinical care: Detecting highly informative events in electronic health records with foundation models, Pacific Symposium on Biocomputing 31 (2026), 173–188 ↩
-
L. Solo, M. McDermott, W. Parker, B. Ramadan, M. Burkhart, & B. Beaulieu-Jones, Efficient generative prediction for EHR foundation models: the SCOPE and REACH estimators, arXiv:2602.03730 ↩
-
I. Lee, L. Solo, M. Burkhart, B. Ramadan, W. Parker, & B. Beaulieu-Jones, Representation before training: a fixed-budget benchmark for generative medical event models, arXiv:2604.16775 ↩
-
M. Burkhart, L. Solo, I. Lee, S. Charles, Z. Liao, K. Chhikara, D. Therese, W.-T. Liao, C. Gao, W. Parker, & B. Beaulieu-Jones, Federated generative event models for tokenized electronic health records, arXiv:2608.02939 ↩