Organise output csv - #508
Merged
Merged
Conversation
-Drop the entity-id suffix from output filenames (Statistics21.csv ->
DemographicStatistics.csv) by exporting singleton collections rather
than bare objects. Row values are unchanged.
-Split Statistics2, which held 51 columns across four domains, and
rename the outputs for what they contain:
Statistics -> WealthIncomeStatistics
Statistics2 -> DemographicStatistics
EmploymentStatistics -> LabourStatistics
AlignmentAdjustmentFactors -> AlignmentStatistics
HealthStatistics -> WellbeingByGender
-HealthStatistics is now a new output for population health by age band.
AgeBandAggregates computes the shared age-band statistics once per year,
so the independently-toggled outputs cannot drift apart.
-Remove 12 calibration residuals reporting simulated minus an outdated
2019 UKHLS target; nothing read them.
-Add OutputReadme, writing README.md beside the CSVs with the run
configuration (population size, years, runs, alignment, bootstrapAll).
-Config keys follow the field renames; old keys are silently ignored.
-Drop the entity-id suffix from output filenames (Statistics21.csv ->
DemographicStatistics.csv) by exporting singleton collections rather
than bare objects. Row values are unchanged.
-Split Statistics2, which held 51 columns across four domains, and
rename the outputs for what they contain:
Statistics -> WealthIncomeStatistics
Statistics2 -> DemographicStatistics
EmploymentStatistics -> LabourStatistics
AlignmentAdjustmentFactors -> AlignmentStatistics
HealthStatistics -> WellbeingByGender
-HealthStatistics is now a new output for population health by age band.
AgeBandAggregates computes the shared age-band statistics once per year,
so the independently-toggled outputs cannot drift apart.
-Remove 12 calibration residuals reporting simulated minus an outdated
2019 UKHLS target; nothing read them.
-Add OutputReadme, writing README.md beside the CSVs with the run
configuration (population size, years, runs, alignment, bootstrapAll).
-Config keys follow the field renames; old keys are silently ignored.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR reorganises SimPaths’ annual CSV outputs by splitting the previous monolithic statistics exports into clearer, topic-specific files (wealth/income, demographics, labour, health, wellbeing, alignment), updating persistence configuration, integration tests, and documentation accordingly, and adding an output-folder README generator to preserve run configuration context alongside results.
Changes:
- Replaces legacy
Statistics*exports with newWealthIncomeStatistics,DemographicStatistics,LabourStatistics,HealthStatistics,WellbeingByGender, andAlignmentStatisticsoutputs. - Updates
SimPathsCollectorscheduling/export flow (including sharedAgeBandAggregates) and adjusts model/caller code to use the new statistics objects. - Refreshes integration tests + expected CSV fixtures and updates repository documentation/configs to reflect the new filenames and semantics.
Reviewed changes
Copilot reviewed 32 out of 32 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/test/java/simpaths/integrationtest/RunSimPathsIntegrationTest.java | Updates integration assertions/comparisons to the new CSV filenames and adds coverage for new outputs. |
| src/test/java/simpaths/integrationtest/expected/WellbeingByGender.csv | New expected output fixture for wellbeing-by-gender export. |
| src/test/java/simpaths/integrationtest/expected/WealthIncomeStatistics.csv | New expected output fixture replacing legacy Statistics1.csv. |
| src/test/java/simpaths/integrationtest/expected/Statistics21.csv | Removes legacy expected fixture superseded by new split outputs. |
| src/test/java/simpaths/integrationtest/expected/Statistics1.csv | Removes legacy expected fixture superseded by WealthIncomeStatistics.csv. |
| src/test/java/simpaths/integrationtest/expected/LabourStatistics.csv | New expected output fixture for labour statistics export. |
| src/test/java/simpaths/integrationtest/expected/HealthStatistics.csv | New expected output fixture for age-band health export. |
| src/test/java/simpaths/integrationtest/expected/EmploymentStatistics1.csv | Removes legacy expected fixture superseded by LabourStatistics.csv. |
| src/test/java/simpaths/integrationtest/expected/DemographicStatistics.csv | New expected output fixture for demographic age-band export. |
| src/test/java/simpaths/integrationtest/expected/AlignmentStatistics.csv | Updates expected fixture to the renamed alignment output. |
| src/test/java/simpaths/data/statistics/LabourStatisticsTest.java | Renames the test class/display name to match the new LabourStatistics naming. |
| src/main/resources/META-INF/persistence.xml | Updates JPA class list to new statistics entities and removes legacy ones. |
| src/main/java/simpaths/model/SimPathsModel.java | Adds getters needed for README/export reporting of model options. |
| src/main/java/simpaths/model/LabourMarket.java | Switches percentile/quintile lookups to the new wealth/income statistics object. |
| src/main/java/simpaths/model/BenefitUnit.java | Switches income-quintile updates to the new wealth/income statistics object. |
| src/main/java/simpaths/experiment/SimPathsMultiRun.java | Adds explicit multirun-mode tracking/accessors used by output README reporting. |
| src/main/java/simpaths/experiment/SimPathsCollector.java | Refactors collection/export pipeline to new stats objects, adds shared AgeBandAggregates, and writes an output README when exporting CSVs. |
| src/main/java/simpaths/experiment/OutputReadme.java | New utility to write README.md beside CSV outputs describing run configuration + files. |
| src/main/java/simpaths/data/statistics/WellbeingByGender.java | New entity producing stacked wellbeing/QoL outputs by gender for ages 25–64. |
| src/main/java/simpaths/data/statistics/WealthIncomeStatistics.java | New entity holding wealth/income distribution and age-band aggregates. |
| src/main/java/simpaths/data/statistics/Statistics2.java | Removes legacy monolithic statistics entity superseded by split outputs. |
| src/main/java/simpaths/data/statistics/Statistics.java | Removes legacy monolithic statistics entity superseded by split outputs. |
| src/main/java/simpaths/data/statistics/LabourStatistics.java | Renames/extends labour statistics entity and adds age-band FT/PT shares (via aggregates). |
| src/main/java/simpaths/data/statistics/HealthStatistics.java | Refactors to age-band health/disability outputs (wellbeing moved to WellbeingByGender). |
| src/main/java/simpaths/data/statistics/DemographicStatistics.java | New entity for demographics by age band. |
| src/main/java/simpaths/data/statistics/AlignmentStatistics.java | Renames alignment adjustment factors entity to align with new output naming. |
| src/main/java/simpaths/data/statistics/AgeBandAggregates.java | New shared computation of age-band aggregates used by multiple annual outputs. |
| documentation/wiki/developer-guide/repository-guide.md | Updates developer documentation to list the new output CSV filenames and meanings. |
| documentation/repository-guide.md | Updates repository guide output-file listing to match new structure. |
| config/test_run.yml | Updates integration test config keys for renamed collector persistence toggles. |
| config/default.yml | Updates collector output documentation/comments to reflect new outputs. |
| CLAUDE.md | Updates architecture/output documentation to reflect new output filenames and shared aggregate approach. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| # The S-Index is an economic (in)security index computed from a rolling window of | ||
| # equivalised consumption, discounted and weighted by a risk-aversion parameter. | ||
| # SIndex_p50 is reported in Statistics1.csv each year. | ||
| # SIndex_p50 is reported in Statistics.csv each year. |
Comment on lines
+100
to
+116
| for (int ii=0; ii<=2; ii++) { | ||
|
|
||
| if (popula[ii]>=0) { | ||
|
|
||
| labInc[ii] /= (workFT[ii] + workPT[ii]); | ||
| prMarr[ii] /= popula[ii]; | ||
| avkids[ii] /= popula[ii]; | ||
| health[ii] /= popula[ii]; | ||
| prDisa[ii] /= popula[ii]; | ||
| workFT[ii] /= popula[ii]; | ||
| workPT[ii] /= popula[ii]; | ||
| invInc[ii] /= popula[ii]; | ||
| penInc[ii] /= popula[ii]; | ||
| invLosses[ii] /= popula[ii]; | ||
| grossDisInc[ii] /= popula[ii]; | ||
| wealth[ii] /= popula[ii]; | ||
| } |
Comment on lines
+877
to
+887
| public WealthIncomeStatistics getWealthIncomeStats() { | ||
| return wealthIncomeStats; | ||
| } | ||
|
|
||
| public void setStats(Statistics stats) { | ||
| this.stats = stats; | ||
| public void setStats(WealthIncomeStatistics wealthIncomeStats) { | ||
| this.wealthIncomeStats = wealthIncomeStats; | ||
| } | ||
|
|
||
| public Statistics2 getStats2() { return stats2; } | ||
| public DemographicStatistics getStats2() { return demographicStats; } | ||
|
|
||
| public void setStats2(Statistics2 stats2) { this.stats2 = stats2; } | ||
| public void setStats2(DemographicStatistics demographicStats) { this.demographicStats = demographicStats; } |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.
Suppressed comments (4)
src/main/java/simpaths/data/statistics/AgeBandAggregates.java:104
- In the age-band normalisation loop,
if (popula[ii] >= 0)is always true (population starts at 0), so this block will divide by zero when an age band is empty. Also,labInc[ii] /= (workFT[ii] + workPT[ii])divides by zero when there are no workers in the band. Guard both denominators so empty bands / zero-worker bands don’t silently produce NaN/Infinity.
for (int ii=0; ii<=2; ii++) {
if (popula[ii]>=0) {
labInc[ii] /= (workFT[ii] + workPT[ii]);
src/main/java/simpaths/experiment/SimPathsCollector.java:883
- The setter name
setStats(...)no longer matches the field/getter (wealthIncomeStats/getWealthIncomeStats()), which makes the public API confusing (and makes reflective tooling harder to follow). Consider keepingsetStatsas a deprecated alias and adding a correctly namedsetWealthIncomeStatsmethod.
public void setStats(WealthIncomeStatistics wealthIncomeStats) {
this.wealthIncomeStats = wealthIncomeStats;
}
src/main/java/simpaths/experiment/SimPathsCollector.java:887
- Similarly,
getStats2()/setStats2(...)now return/acceptDemographicStatistics, so the method names are misleading. Adding clearly named accessors (and leaving the old names as deprecated aliases) will reduce confusion and make future refactors safer.
public DemographicStatistics getStats2() { return demographicStats; }
public void setStats2(DemographicStatistics demographicStats) { this.demographicStats = demographicStats; }
config/default.yml:46
- This comment still references
Statistics.csv, but the S-Index is now written toWealthIncomeStatistics.csv. Updating this avoids misleading configuration documentation.
# SIndex_p50 is reported in Statistics.csv each year.
Comment on lines
+124
to
+126
| public void setGender(String demSex) { | ||
| this.demSex = demSex; | ||
| } |
- Guard both denominators in AgeBandAggregates: empty age bands and
bands with no workers no longer divide by zero. Non-empty bands are
unchanged.
- Give WellbeingByGender a distinct entity id per gender group
(Total=1, Male=2, Female=3); the constant id collided on the JPA
primary key across the three rows written each year.
- Rename setStats/getStats2/setStats2 to match their fields.
- Fix stale Statistics.csv reference in default.yml.
…ender.java Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.