Fix the inverted EDUCQUAL to highest_education mapping - #460
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
vahid-ahmadi
left a comment
There was a problem hiding this comment.
Verified against both sides of its provenance: the corrected map is identical, code for code, to the frame the microcosm migration adjudicated and verified in PolicyEngine/microcosm#703 — where I checked it against the harmonised codeframe during that review — and the per-code labels here match the SN 9367 dictionary labels quoted in the body (1 = Doctorate or MPhil, 17 = A-Level or equivalent, 18/19 the near-empty baccalaureates, 36–49 the GCSE band, 83–85 entry level). The two-way evidence — dictionary labels verbatim plus the disclosure-safe aggregates (code 1 at 1.8% of adults with top mean earnings is not "not completed primary") — is exactly the standard a coded-domain correction should meet, and enumerating every code with its label instead of the old range-loops means the next dictionary revision gets caught by a diff rather than absorbed by a for code in range(...).
The composition move ({TERTIARY 9.9%, POST_SECONDARY 21.3%} → {28.1%, 8.4%}) lands the base FRS at a plausible UK degree-holder share for the first time, and flagging the expected downstream shifts in student-support outputs up front — rather than letting the next data build discover them — is the right disclosure. Keeping the two pipelines on the same frame is itself worth landing this promptly, since every microcosm head-to-head from E3 onward diffs against this repo's output.
Two small things:
- The new comment header says "Based on FRS 2022-23 documentation (SPSS value labels)" while the changelog and PR body cite the FRS 2023-24 dictionary (SN 9367). If the map was verified against 2023-24 — which the body's evidence says it was — the comment should name that vintage (or both, if the 2022-23 labels were checked too). After the migration's region-map lesson, the vintage a coded-domain map cites is load-bearing documentation.
- Code 87's ~13% of adults defaulting to UPPER_SECONDARY is preserved behavior and correctly out of scope here, but it's now the single largest source of
highest_educationuncertainty — an eighth of adults assigned by fallback. Worth a follow-up issue so it doesn't hide behind the fixed map: even a crude imputation from age/earnings would beat a constant default for a bucket that size.
With the vintage comment fixed, this is ready to land as far as I'm concerned.
Fixes #459.
The
EDUCQUAL_MAPread the FRS highest-qualification codeframe upside down: code 1 was mapped to NOT_COMPLETED_PRIMARY and codes 17–21 to TERTIARY, but the frame starts at 1 = Doctorate and descends.highest_educationfeeds live model logic (maintenance loan system assignment, DSA course eligibility, student loan plan assignment).This cherry-picks the existing correction from
feat/impute-below-threshold-borrowers(8f95629, authorship preserved) onto main, now verified two independent ways:Under the corrected map the weighted
highest_educationcomposition of the base FRS moves from {TERTIARY 9.9%, POST_SECONDARY 21.3%} to {TERTIARY 28.1%, POST_SECONDARY 8.4%} — expect downstream shifts in student-support and education-conditioned outputs on the next data build.The microcosm migration adopted the same corrected frame as a signed difference in PolicyEngine/microcosm#703, so landing this keeps the two pipelines aligned.
Test suite: 484 passed, 107 skipped (licensed-data tests skip without staged inputs), no failures.
🤖 Generated with Claude Code