Update of main (pre-September26-release) - #518
Merged
Merged
Conversation
Deleted old validation plot files and uploaded new validation do files.
Persistence.createEntityManagerFactory(...) was called on every invocation of all three methods. Building a factory is expensive — it validates the schema and sets up a connection pool. Two static fields replace this: - emfStartingPopulationPersist — shared by getProcessed() and persistProcessed() (both use getPersistDatabasePath()) - emfStartingPopulationRun — used by loadStartingPopulation() (uses RunDatabasePath) 2. loadStartingPopulation() — bug fix for LazyInitializationException With @onetomany collections already marked FetchType.LAZY, returning detached Household objects after em.close() without having initialised benefitUnits or members would throw LazyInitializationException the moment the caller iterated them. The force-initialisation loop triggers the SUBSELECT batch loads within the open session before closing it. 3. getProcessed() — dead variable removed processed_return was assigned the JOIN FETCH result but the method returned processed (the same L1-cached object). The two are identical references, so processed_return.resetDependents() and processed.resetDependents() were equivalent. The dead variable is removed and the call is chained directly. No behaviour change.
-DonorTaxImputation used Math.random() to decide whether a benefit unit receives Universal Credit based on the weighted mean UC receipt across donor candidates. Math.random() draws from Java's global, unseeded Random instance, producing different results on every JVM run regardless of the model's random seed setting. -Replaced with SimulationEngine.getRnd().nextDouble(), which draws from the model's seeded Random instance — the same one used throughout the rest of the model. With a fixed seed, UC/Legacy benefit assignment is now deterministic across runs.
AlignmentAdjustmentFactors fields (30 total): - Partnership: adj factor, sim share, target share (same computation) - Fertility: adj factor, sim rate, target rate (same computation) - In-school: adj factor, sim share, target share — new - Utility adj factors × 7 (same computation) - Employment shares: 7 × simulated + 7 × target — new
Optimized database hadling and replaced Statistics3 with AlignmentAdjustmentFactors
Remove all the code/references to IT
Remove all remaining hard-coded values (issue #393)
Replace <h3> with <h2> to solve error in PR #453.
Introduces a new model parameter `taxDonorUpratingByWage` that, when enabled, scales simulated household income by the real wage growth index (TimeSeriesVariable.WageGrowth) before nearest-neighbour matching against the tax donor database. When disabled (default), the existing price-growth-only behaviour is preserved. In both cases, imputed financial flows are deflated back to BASE_PRICE_YEAR (2015) using price inflation only (TimeSeriesVariable.Inflation). The option is exposed as a JAS-mine @GUIparameter checkbox in SimPathsModel, is configurable via YAML (taxDonorUpratingByWage), and is written to the per-run config output file. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Stop applying regression RMSE and random Gaussian draws to the childcare cost (C1b) regression. In the latest estimate update, C1b was no longer included in RMSE. The old value lingered for the sake of code functioning. With this commit, the code is updated accordingly.
Tags to initialise attributes
The wrapping `AccumulatorStats` is unnecessary as we're only interested in the last value.
Use new `Supplier`-based jasmine API
This also removes obsolete statistic fixture files that are no longer generated by the updated output set.
Switch hours of care provided to ordered logit
Apply the Alignment module naming convention to all 30 variables in AlignmentStatistics: fields, accessors and @column names now carry the align prefix (e.g. shareCohabitingSim -> alignPartnerSimShare). CSV columns are sorted alphabetically by field name, so the headers in AlignmentStatistics.csv are both renamed and reordered; the expected integration-test file is updated to match, with values unchanged.
…Statistics Rename AlignmentStatistics variables to align* convention
This commit restores the default simulation configuration to a shorter 2019-2022 run with a 50k population. It updates the codebook spreadsheet inputs. It also cleans up the wealth statistics schema by removing the legacy equivalised disposable income alias and renaming the S-index median field to match the codebook. Obsolete partner education enum entries were removed from Person, and the collector logic was updated to stop writing the legacy median-income output.
Drops the unused `EDI` collector wiring from `SimPathsCollector` and removes the legacy `edi_p50` output alias. The integration-test expected `WealthIncomeStatistics.csv` header is updated accordingly, including the standardized `statSIndexP50` naming and revised column layout to match current codebook-aligned output.
Various updates
Contributor
There was a problem hiding this comment.
Pull request overview
This PR is a broad pre-release update that modernises the validation/tooling scripts and refactors SimPaths’ statistics, filtering, and persistence outputs (including updated integration-test expectations) to match the current UK-focused model output structure.
Changes:
- Updates validation Stata
.dofiles with clearer headers/notes, consistent “shaded area” explanations, and optional${min_sim_year}/${max_sim_year}controls. - Renames/restructures persisted statistics outputs (e.g.,
WealthIncomeStatistics.csv,DemographicStatistics.csv,LabourStatistics.csv) and aligns integration tests + expected CSV fixtures accordingly. - Refactors multiple model/data components (filters, enums, social care constructs, donor-tax tooling) and updates documentation and CI workflows to reflect the current setup.
Reviewed changes
Copilot reviewed 115 out of 247 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| validation/02_simulated_output_validation/do_files/04_17_plot_disability.do | Adds descriptive header and clearer graph notes |
| validation/02_simulated_output_validation/do_files/04_14_plot_at_risk_of_poverty.do | Adds year bounds overrides and clarifies shaded-band note |
| validation/02_simulated_output_validation/do_files/04_12_plot_partnership_status.do | Adds description and consistent shaded-band notes |
| validation/02_simulated_output_validation/do_files/04_11_plot_income_shares.do | Cleans notes/typos and fixes output filenames |
| validation/02_simulated_output_validation/do_files/04_10_1_plot_hours_worked_discrete.do | Improves description and legend wording |
| validation/02_simulated_output_validation/do_files/04_08_plot_equivalised_disposable_income.do | Adds year bounds overrides and shaded-band note |
| validation/02_simulated_output_validation/do_files/04_07_plot_disposable_income.do | Adds year bounds overrides and note cleanup |
| validation/02_simulated_output_validation/do_files/04_06_plot_pension_income.do | Clarifies sample selection + shaded-band notes + year overrides |
| validation/02_simulated_output_validation/do_files/04_05_plot_capital_income.do | Adds description, shaded-band notes, year overrides |
| validation/02_simulated_output_validation/do_files/04_04_plot_gross_labour_income.do | Adds description, shaded-band notes, year overrides |
| validation/02_simulated_output_validation/do_files/03_create_UKHLS_validation_targets.do | Improves documentation and fixes label name typo |
| validation/02_simulated_output_validation/do_files/02_create_simulated_variables.do | Improves documentation header |
| validation/02_simulated_output_validation/do_files/01_prepare_simulated_data.do | Clarifies inputs and switches to ${max_sim_year} |
| src/test/java/simpaths/integrationtest/RunSimPathsIntegrationTest.java | Updates expected output filenames and adds numeric tolerances |
| src/test/java/simpaths/integrationtest/expected/WealthIncomeStatistics.csv | Adds new expected wealth/income statistics fixture |
| src/test/java/simpaths/integrationtest/expected/Statistics31.csv | Removes obsolete expected statistics fixture |
| src/test/java/simpaths/integrationtest/expected/Statistics1.csv | Removes obsolete expected statistics fixture |
| src/test/java/simpaths/integrationtest/expected/LabourStatistics.csv | Adds new expected labour statistics fixture |
| src/test/java/simpaths/integrationtest/expected/HealthStatistics.csv | Adds new expected health statistics fixture |
| src/test/java/simpaths/integrationtest/expected/EmploymentStatistics1.csv | Removes obsolete expected employment fixture |
| src/test/java/simpaths/integrationtest/expected/DemographicStatistics.csv | Adds new expected demographic statistics fixture |
| src/test/java/simpaths/data/statistics/LabourStatisticsTest.java | Updates unit test to new LabourStatistics API |
| src/test/java/simpaths/data/ParametersTest.java | Adds regression regressor-validation test |
| src/test/java/simpaths/data/filters/EmploymentHistoryFilterTest.java | Migrates to predicate-based filter API |
| src/main/resources/META-INF/persistence.xml | Updates persisted entity list to new statistics entities |
| src/main/java/simpaths/model/UnionMatching.java | Switches to explicit matching class instantiation |
| src/main/java/simpaths/model/taxes/TestTaxRoutine.java | Updates hard-coded tax test scenarios |
| src/main/java/simpaths/model/taxes/Matches.java | Updates commons-csv builder usage |
| src/main/java/simpaths/model/taxes/KeyFunction.java | Renames disability fields for clarity/consistency |
| src/main/java/simpaths/model/taxes/DonorTaxUnitPolicy.java | Adds normalised-income getter overload |
| src/main/java/simpaths/model/taxes/DonorPerson.java | Renames disability getter for clarity |
| src/main/java/simpaths/model/taxes/database/TaxDonorDataParser.java | Improves typing and renames disability getter usage |
| src/main/java/simpaths/model/taxes/database/MatchIndicesSet.java | Updates commons-csv builder usage |
| src/main/java/simpaths/model/taxes/database/InputDataSet.java | Adds generics and updates commons-csv builder usage |
| src/main/java/simpaths/model/lifetime_incomes/ManagerProjectLifetimeIncomes.java | Updates logger import and generics |
| src/main/java/simpaths/model/InitialPopulationFilter.java | Updates partner-status accessor usage |
| src/main/java/simpaths/model/Household.java | Migrates logging + renames getter for original HH id |
| src/main/java/simpaths/model/HibernateUtil.java | Formatting-only brace adjustment |
| src/main/java/simpaths/model/FertilityAlignment.java | Makes fertility alignment objective deterministic/smooth |
| src/main/java/simpaths/model/enums/TimeSeriesVariable.java | Removes obsolete enum constant |
| src/main/java/simpaths/model/enums/SocialCareProvision.java | Removes old multi-category social-care provision enum |
| src/main/java/simpaths/model/enums/Region.java | Removes Italy region entries (UK-only focus) |
| src/main/java/simpaths/model/enums/Education.java | Changes integer-coded ordering and adds rank helper |
| src/main/java/simpaths/model/enums/Country.java | Removes Italy (UK-only focus) |
| src/main/java/simpaths/model/enums/CareHoursProvidedCategory.java | Adds ordered care-hours category enum |
| src/main/java/simpaths/model/decisions/States.java | Simplifies social-care provision state representation |
| src/main/java/simpaths/model/decisions/ManagerFileGrids.java | Updates commons-csv builder usage |
| src/main/java/simpaths/model/decisions/GridScale.java | Adjusts social-care provision axis cardinality |
| src/main/java/simpaths/model/decisions/ExpectationsFactory.java | Updates accessors and social-care expectations logic |
| src/main/java/simpaths/model/decisions/DecisionTests.java | Updates commons-csv builder usage |
| src/main/java/simpaths/model/annotations/NullInitialised.java | Adds new period-reset annotation |
| src/main/java/simpaths/model/annotations/Lag.java | Adds new lag-copy annotation |
| src/main/java/simpaths/data/TestRegressions.java | Updates Person proxy setter names |
| src/main/java/simpaths/data/statistics/Statistics.java | Removes legacy statistics entity |
| src/main/java/simpaths/data/statistics/EmploymentStatistics.java | Removes legacy employment statistics entity |
| src/main/java/simpaths/data/statistics/DemographicStatistics.java | Adds new demographic statistics entity |
| src/main/java/simpaths/data/startingpop/DataParser.java | Removes old social-care provision column logic |
| src/main/java/simpaths/data/ScenarioTable.java | Adds generic type to column class |
| src/main/java/simpaths/data/RegressionName.java | Switches S3c/S3d to ordered-logit regression type |
| src/main/java/simpaths/data/ManagerRegressions.java | Moves S3c/S3d to ordered regression + fixes event list creation |
| src/main/java/simpaths/data/filters/ValidPersonEarningsCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/ValidHouseholdIncomeRegionalCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/ValidHouseholdIncomeCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/ValidHomeownersCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/ValidEducationRegionCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/ValidEducationAgeGroupCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/RegionEducationWorkingCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/RegionEducationCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/RegionEducationAtRiskOfWorkCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/RegionCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/MaleRegionCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/MaleRegionAgeCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/MaleAgeGroupEducationCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/MaleAgeGroupCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/GenderWorkingCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/GenderEducationWorkingCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/GenderEducationCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/GenderCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/FlexibleInLabourSupplyPersonFilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/FlexibleInLabourSupplyFilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/FlexibleInLabourSupplyByEducationFilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/FlexibleInLabourSupplyByAgeAndGenderFilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/Filters.java | Adds consolidated predicate-based Person filters |
| src/main/java/simpaths/data/filters/FertileFilter.java | Removes legacy fertility predicate |
| src/main/java/simpaths/data/filters/FemaleWithChildrenByAgeBandFilter.java | Leaves legacy filter commented-out (no functional change) |
| src/main/java/simpaths/data/filters/FemalesWithoutChildrenAgeGroupCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/FemalesWithChildrenByChildAgeCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/FemalesWithChildrenAgeGroupEducationCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/FemalesWithChildrenAgeGroupCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/FemalesUnder45Filter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/FemalesAgeGroupEducationCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/FemaleRegionCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/FemaleRegionAgeCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/FemaleCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/FemaleAgeGroupEducationCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/FemaleAgeGroupCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/EmploymentHistoryFilter.java | Removes legacy filter class (moved to Filters) |
| src/main/java/simpaths/data/filters/EducationEmployedCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/EducationCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/ChildValidIncomeRegionalCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/ChildValidIncomeCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/CanBePartneredCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/BenefitUnitFilters.java | Adds consolidated predicate-based BenefitUnit filters |
| src/main/java/simpaths/data/filters/AgeGroupCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/filters/AgeGenderCSfilter.java | Removes legacy filter |
| src/main/java/simpaths/data/ComboBoxYear.java | Fixes JComboBox scoping and event handling |
| src/main/java/simpaths/data/ComboBoxCountry.java | Updates to UK-only defaults and static lookup usage |
| input/InitialPopulations/compile/RegressionEstimates/04_reg_fertility.do | Adds refactored fertility regression estimation script |
| input/InitialPopulations/compile/RegressionEstimates/02_reg_leave_parental_home.do | Adds refactored leave-parental-home estimation script |
| input/InitialPopulations/compile/RegressionEstimates/00_master_regression_estimates.do | Adds master runner for regression-estimates pipeline |
| input/InitialPopulations/compile/99_training_data.do | Extends wealth training features and end year |
| input/InitialPopulations/compile/08_wealth_to_ukhls.do | Adds unsecured debt and pension contribution rates to imputation |
| input/InitialPopulations/compile/04_social_care_provided.do | Adds weighted reallocation of care-hours categories 8/9 |
| input/InitialPopulations/compile/03_social_care_received.do | Centralises wage-rate matrix usage (via master) and loops costs |
| input/InitialPopulations/compile/02_create_UKHLS_variables.do | Switches CPI deflators to inflation matrix and adds debt fields |
| input/InitialPopulations/compile/00_master.do | Adds inflation + care-wage matrices and updates wealth end year |
| input/DoFilesTarget/03b_calculate_partnership_target.do | Renames targets workbook and embeds metadata sheet |
| input/DoFilesTarget/03a_calculate_partneredShare_initialPop_BUlogic.do | Renames targets workbook and embeds metadata sheet |
| input/DoFilesTarget/02_inSchool_targets_initpopdata.do | Renames targets workbook and embeds metadata sheet |
| input/DoFilesTarget/01_employment_shares_initpopdata.do | Renames targets workbook and embeds metadata sheet |
| documentation/wiki/overview/parameterisation.md | Updates alignment-targets file list/naming |
| documentation/wiki/getting-started/index.md | Simplifies getting-started landing page |
| documentation/wiki/getting-started/environment-setup.md | Updates stated toolchain requirements |
| documentation/wiki/developer-guide/working-in-github.md | Points requirements to environment setup |
| documentation/wiki/developer-guide/repository-guide.md | Updates repo guide for new output/statistics files |
| documentation/wiki/developer-guide/internals/multirun-implementation.md | Points prerequisites to environment setup |
| documentation/wiki/developer-guide/internals/api.md | Removes embedded workflow YAML excerpts |
| documentation/repository-guide.md | Updates repo guide for new output/statistics files |
| documentation/flowcharts/flowchart_review_prompt.md | Adds flowchart review prompt artifact |
| config/test_run.yml | Updates collector persistence flags to new statistics names |
| .github/workflows/SimPathsBuild.yml | Updates CI Java version and action versions |
| .github/workflows/publish-javadoc.yml | Updates Java version and action versions |
| .github/workflows/deploy-docs.yml | Updates action versions for docs deployment |
| .claude/settings.local.json | Adds local Claude tool permissions config |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+444
to
+447
| forvalues yy = $careWageRate_minyear/$careWageRate_maxyear { | ||
|
|
||
| replace formal_socare_cost = formal_socare_hrs * careHourlyWageRates[`yy'-${careWageRate_minyear},1] if (formal_socare_hrs>0) & stm==`yy' | ||
| } |
Collaborator
Author
There was a problem hiding this comment.
@justin-ven or @dariaple can you please check this?
Comment on lines
+231
to
+235
| global inflation_minyear = 2008 | ||
| global inflation_maxyear = 2025 | ||
| matrix inflation = ( /// | ||
| 0.862 \ /// 2009 | ||
| 0.879 \ /// 2009 |
Comment on lines
+215
to
216
| note("Note: Poverty line calculated within each year as 60% of the median equivalised disposable income of benefit unit. Calculated" "using individual level observations.Shaded area = mean +/- 1.96*SD across $max_n_runs simulation runs.", /// | ||
| size(vsmall)) |
Contributor
|
Looks good to me |
justin-ven
self-requested a review
August 31, 2026 07:36
justin-ven
approved these changes
Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.