fix(mimic-iv): retain device-only rows in oxygen_delivery - #2071
Conversation
The FULL OUTER JOIN of O2 flow and delivery devices filtered with WHERE ce.rn = 1, which dropped rows where only a device (itemid 226732) was charted. Filter rn in ce_stg3 before the join so device-only rows are retained. Regenerate postgres and duckdb dialect copies. Fixes MIT-LCP#1570
Chessing234
left a comment
There was a problem hiding this comment.
the post-join where ce.rn = 1 was definitely nuking device-only rows from the full outer join — moving the rn filter into ce_stg3 first is the right shape. coalesce on subject/stay/charttime was already there so device-only keys should survive. if you haven't already, worth a quick count of rows where o2_flow is null but o2_delivery_device_1 isn't, before vs after.
|
keeping device-only oxygen rows is useful for vent studies. does the regenerated postgres/duckdb copy land in this pr too? |
Chessing234
left a comment
There was a problem hiding this comment.
filtering rn before the full outer join is the right fix — post-join where ce.rn = 1 was dropping device-only o2 rows when ce was null. mirrored across dialects; the comment in the bq file helps. a tiny regression query in concepts/documentation (device-only count before/after) would make the behavioral change obvious to reviewers.
Summary
rnince_stg3before the FULL OUTER JOIN inoxygen_delivery.sqlWHERE ce.rn = 1ran after the joinTest plan
pytest tests/test_transpile.py -k oxygen_deliverypasses locallyoxygen_deliveryFULL OUTER JOINs O2 flow (ce_stg2) with delivery devices (o2, itemid 226732), then appliedWHERE ce.rn = 1. That filter drops rows whereceis NULL, so stays with device charting but no flow (e.g.stay_id = 30813912in #1570) never appear in the derived table. Move thern = 1filter intoce_stg3before the join so device-only rows are kept. Regenerate postgres and duckdb dialect copies from the BigQuery source.Fixes #1570