Organise output csv

CLAUDE.md

@@ -88,7 +88,7 @@ All behavioural equations are estimated externally and loaded at startup from Ex
 
 ### Alignment
 
-Several demographic processes are aligned to external targets (ONS projections, LFS shares) via `ActivityAlignmentV2`, `FertilityAlignment`, `PartnershipAlignment`, `InSchoolAlignment`, `SocialCareAlignment`. Alignment factors are exported to `AlignmentAdjustmentFactors1.csv` each run.
+Several demographic processes are aligned to external targets (ONS projections, LFS shares) via `ActivityAlignmentV2`, `FertilityAlignment`, `PartnershipAlignment`, `InSchoolAlignment`, `SocialCareAlignment`. Alignment factors are exported to `AlignmentStatistics.csv` each run.
 
 ### Configuration
 
@@ -97,10 +97,14 @@ Runs are configured via YAML files in `config/`. `default.yml` documents all ava
 ### Output
 
 Output is written to `output/<run-timestamp>/csv/` by `SimPathsCollector`. Key files:
-- `Statistics1.csv` — income distribution (Gini, percentiles, S-Index)
-- `Statistics2.csv` — demographic validation (partnership, employment, health by age/gender)
-- `EmploymentStatistics.csv`, `HealthStatistics.csv` — domain-specific time series
-- `AlignmentAdjustmentFactors1.csv` — alignment diagnostics
+- `WealthIncomeStatistics.csv` — income and wealth (Gini, percentiles, S-Index, plus income and wealth by age band)
+- `DemographicStatistics.csv` — demographics by age band (partnership, dependent children, population counts)
+- `HealthStatistics.csv` — population health by age band (self-rated health, disability shares)
+- `LabourStatistics.csv` — labour market transitions, participation, full-time/part-time shares by age band
+- `AlignmentStatistics.csv` — alignment diagnostics
+- `WellbeingByGender.csv` — wellbeing and quality of life (SF-12, GHQ-12, EQ-5D), ages 25–64, three rows per year (Total/Male/Female)
+
+The four age-band outputs share one population traversal per year via `AgeBandAggregates`; the bands are 18–29, 30–54 and 55–74.
 
 ### Integration tests
 

config/default.yml

@@ -1,8 +1,18 @@
 # SimPaths multi-run configuration file.
 # Uncomment and edit any field to override its default value.
 # CLI flags take final precedence over anything set here.
+#
+# Commented-out keys show the class default; uncommenting one overrides it.
+# An unrecognised key is not an error - it logs a stack trace and is ignored,
+# so a mistyped or renamed key silently leaves the default in place.
+#
+# Only int, long, boolean, double and String fields can be set from this file.
+# Fields of other types (for example the UnionMatchingMethod enum behind union
+# matching) are not settable here and will fail at startup if added.
 
-# ── Top-level run arguments ────────────────────────────────────────────────────
+# ── Top-level run arguments ───────────────────────────────────────────────────
+# Unlike the sections below, these are active settings rather than documented
+# defaults - they override the SimPathsMultiRun defaults on every run.
 
 maxNumberOfRuns: 1        # number of sequential simulation runs
 executeWithGui: false     # true = launch JAS-mine GUI; false = headless (required on servers/CI)
@@ -23,16 +33,17 @@ model_args:
   # --- Time trend controls ---
 #    maxAge: 130                              # maximum age kept in simulation; persons above this are removed
 #    fixTimeTrend: true                       # if true, freezes the time trend in regression equations
-#    timeTrendStopsIn: 2017                   # year at which the time trend is frozen (if fixTimeTrend: true)
-#    timeTrendStopsInMonetaryProcesses: 2017  # same freeze year applied to monetary/income regressions only
+#    timeTrendStopsIn: 2023                   # year at which the time trend is frozen (if fixTimeTrend: true)
+#    timeTrendStopsInMonetaryProcesses: 2023  # same freeze year for monetary/income regressions;
+                                             # defaults to whatever timeTrendStopsIn is set to
 
   # --- Random number controls ---
 #    fixRandomSeed: true                      # if true, each run uses the same fixed seed (randomSeedIfFixed)
 
   # --- Income security (S-Index) ---
   # The S-Index is an economic (in)security index computed from a rolling window of
   # equivalised consumption, discounted and weighted by a risk-aversion parameter.
-  # SIndex_p50 is reported in Statistics1.csv each year.
+  # SIndex_p50 is reported in WealthIncomeStatistics.csv each year.
 #    sIndexTimeWindow: 5      # length of rolling window in years (default 5)
 #    sIndexAlpha: 2           # coefficient of relative risk aversion (higher = more sensitivity to drops)
 #    sIndexDelta: 0.98        # annual discount factor applied to past consumption observations
@@ -48,18 +59,21 @@ model_args:
 #    useWeights: false        # if true, apply survey weights in alignment and statistics calculations
 
   # --- Matching method ---
-#    useSBAMMatching:         # if true, use SBAM instead of standard union-matching algorithm
+  # Union matching is controlled by the UnionMatchingMethod enum field
+  # `unionMatchingMethod` (default ParametricNoRegion; alternatives Parametric, SBAM).
+  # It cannot be set from this file - enum fields are not supported by the config
+  # loader - so change it in SimPathsModel or via the GUI.
 
   # --- Demographic projections ---
 #    projectMortality: true   # if false, disables stochastic mortality (population does not die)
 
   # --- Alignment flags ---
   # See model-concepts.md for a full explanation of what alignment does.
 #    alignPopulation: true        # align age-sex-region totals to official population projections
-#    alignFertility: true         # scale birth probabilities to match projected fertility rates
+#    alignFertility: false        # scale birth probabilities to match projected fertility rates
 #    alignEducation: false        # align completed education distribution to targets
 #    alignInSchool: false         # align school participation rate (age 16–29) to targets
-#    alignCohabitation: false     # align share of cohabiting individuals to targets
+#    alignCohabitation: true      # align share of cohabiting individuals to targets
 #    alignEmployment: false       # align employment share to targets
 
   # --- Labour market modules ---
@@ -68,7 +82,7 @@ model_args:
 
   # --- Social care and childcare ---
 #    projectFormalChildcare: true          # simulate formal childcare costs
-#    projectSocialCare: false              # simulate social care receipt and provision module
+#    projectSocialCare: true               # simulate social care receipt and provision module
 #    flagSuppressChildcareCosts: false     # if true, set formal childcare costs to zero (scenario use)
 #    flagSuppressSocialCareCosts: false    # if true, set social care costs to zero (scenario use)
 
@@ -94,12 +108,12 @@ model_args:
   # Enables backward-induction life-cycle solution for consumption and labour supply.
   # Decision grids are pre-computed in year 0; agents look up optimal choices each year.
   # Computationally intensive — disabled by default.
-#    enableIntertemporalOptimisations: true
+#    enableIntertemporalOptimisations: false
 
   # IO state-space: which characteristics agents respond to when choosing labour/consumption.
   # Each flag adds a dimension to the grid and increases solve time.
 #    responsesToHealth: true          # include physical health in IO state space
-#    responsesToDisability: false     # include disability status in IO state space
+#    responsesToDisability: true      # include disability status in IO state space
 #    responsesToEducation: true       # include student and education level in IO state space
 #    responsesToPension: false        # include private pension wealth in IO state space
 #    responsesToRetirement: false     # include retirement state (and private pension) in IO state space
@@ -122,6 +136,19 @@ model_args:
   # --- Population load ---
 #    ignoreTargetsAtPopulationLoad: false  # if true, skip alignment-target checks when loading the initial population
 
+  # --- Lifetime income projection ---
+  # Optional module that projects lifetime income paths for synthetic birth cohorts,
+  # run once at model build time before the simulation starts. Off by default.
+#    lifetimeIncomeGenerate: false     # generate a new set of lifetime incomes
+#    lifetimeIncomeStartBirthYear:     # first birth cohort to project (no default; required if generating)
+#    lifetimeIncomeEndBirthYear:       # last birth cohort to project (no default; required if generating)
+#    lifetimeIncomeEndAge:             # age at which each cohort's projection stops (no default)
+#    lifetimeIncomeCohortSize: 10000   # number of synthetic individuals per birth cohort
+#    lifetimeIncomeWriteToCSV: false   # write the projected lifetime incomes to CSV
+#    lifetimeIncomeRandomSeed: 505     # seed for the lifetime income projection
+#    lifetimeIncomeAge0StdDev: 0.9     # standard deviation of the initial income draw at age 0
+#    lifetimeIncomeImpute: false       # impute lifetime income for the initial population at load
+
 
 # ── innovation_args: parameter variation across sequential runs ────────────────
 # These flags control how parameters change between run 0, run 1, run 2, etc.
@@ -145,24 +172,31 @@ innovation_args:
 # ── collector_args: output collection and export ───────────────────────────────
 # Controls what SimPathsCollector writes to CSV / database each year.
 #
-# Output files:
-#   Statistics1.csv    — income distribution: Gini coefficients, income percentiles, median EDI, S-Index
-#   Statistics2.csv    — demographic validation: partnership rates, employment, health, disability by age/gender
-#   AlignmentAdjustmentFactors1.csv — alignment diagnostics: factors, simulated shares, and target shares
-#   EmploymentStatistics.csv — labour market transitions and participation rates
-#   HealthStatistics.csv     — health measures (SF-12, GHQ-12, EQ-5D) by age/gender
+# Output files (one row per year unless noted):
+#   WealthIncomeStatistics.csv — income and wealth: Gini coefficients, income percentiles,
+#                        median EDI, S-Index, and income/wealth by age band (18-29, 30-54, 55-74)
+#   DemographicStatistics.csv  — demographics by age band: partnership rates, dependent children,
+#                        population counts
+#   HealthStatistics.csv       — population health by age band: self-rated health and disability shares
+#   LabourStatistics.csv       — labour market transitions, participation rates, and
+#                        full-time/part-time shares by age band
+#   AlignmentStatistics.csv    — alignment diagnostics: factors, simulated shares, and target shares
+#   WellbeingByGender.csv      — wellbeing and health-related quality of life (SF-12, GHQ-12, EQ-5D),
+#                        ages 25-64, three rows per year: Total, Male, Female
 
 collector_args:
 #    calculateGiniCoefficients: false   # compute Gini coefficients (also populates GUI charts); off by default for speed
 #    exportToDatabase: false            # write outputs to H2 database (in addition to or instead of CSV)
 #    exportToCSV: true                  # write outputs to CSV files under output/<run>/csv/
-#    persistStatistics: true            # write Statistics1.csv (income distribution)
-#    persistStatistics2: true           # write Statistics2.csv (demographic validation outputs)
-#    persistAlignmentAdjustmentFactors: true # write AlignmentAdjustmentFactors1.csv
-#    persistPersons: false              # write one row per person per year (large files)
-#    persistBenefitUnits: false         # write one row per benefit unit per year (large files)
-#    persistHouseholds: false           # write one row per household per year
-#    persistEmploymentStatistics: false # write EmploymentStatistics.csv
+#    persistWealthIncomeStatistics: true      # write WealthIncomeStatistics.csv (income and wealth)
+#    persistDemographicStatistics: true       # write DemographicStatistics.csv (demographics by age band)
+#    persistAlignmentStatistics: true         # write AlignmentStatistics.csv (alignment diagnostics)
+#    persistLabourStatistics: true            # write LabourStatistics.csv (labour market outcomes)
+#    persistHealthStatistics: true            # write HealthStatistics.csv (population health by age band)
+#    persistWellbeingByGender: true           # write WellbeingByGender.csv (wellbeing by gender, ages 25-64)
+#    persistPersons: true               # write one row per person per year (large files)
+#    persistBenefitUnits: true          # write one row per benefit unit per year (large files)
+#    persistHouseholds: true            # write one row per household per year (large files)
 #    dataDumpStartTime: 0L              # first year to write output (0 = startYear)
 #    dataDumpTimePeriod: 1.0            # output frequency in years (1.0 = every year)
 

config/test_run.yml

@@ -10,9 +10,9 @@ popSize: 20000
 integrationTest: true
 
 collector_args:
-    persistStatistics: true
-    persistStatistics2: true
-    persistAlignmentAdjustmentFactors: true
+    persistWealthIncomeStatistics: true
+    persistDemographicStatistics: true
+    persistAlignmentStatistics: true
     persistPersons: false
     persistBenefitUnits: false
     persistHouseholds: false

documentation/repository-guide.md

@@ -64,12 +64,15 @@ SimPaths/
 ├── output/                         # Simulation outputs
 │   ├── [timestamp]_[seed]_[run]/   # Timestamped output folders
 │   │   ├── csv/
-│   │   │   ├── Statistics1.csv          # Income distribution, Gini, S-Index
-│   │   │   ├── Statistics2<N>.csv       # Demographics by age and gender
-│   │   │   ├── Statistics3<N>.csv       # Alignment diagnostics
-│   │   │   ├── Person<N>.csv            # Person-level output
-│   │   │   ├── BenefitUnit<N>.csv       # Benefit-unit-level output
-│   │   │   └── Household<N>.csv         # Household-level output
+│   │   │   ├── WealthIncomeStatistics.csv  # Income and wealth; Gini, S-Index
+│   │   │   ├── DemographicStatistics.csv   # Demographics by age band
+│   │   │   ├── AlignmentStatistics.csv     # Alignment diagnostics
+│   │   │   ├── LabourStatistics.csv        # Labour market transitions and participation
+│   │   │   ├── HealthStatistics.csv        # Population health by age band
+│   │   │   ├── WellbeingByGender.csv       # Wellbeing by gender, ages 25-64
+│   │   │   ├── Person.csv               # Person-level output
+│   │   │   ├── BenefitUnit.csv          # Benefit-unit-level output
+│   │   │   └── Household.csv            # Household-level output
 │   │   ├── database/                    # Run-specific persistence output
 │   │   └── input/                       # Copied run input artifacts
 │   └── logs/                       # Log files (with -f flag on multirun)

documentation/wiki/developer-guide/repository-guide.md

@@ -69,12 +69,15 @@ SimPaths/
 ├── output/                         # Simulation outputs
 │   ├── [timestamp]_[seed]_[run]/   # Timestamped output folders
 │   │   ├── csv/
-│   │   │   ├── Statistics1.csv          # Income distribution, Gini, S-Index
-│   │   │   ├── Statistics2<N>.csv       # Demographics by age and gender
-│   │   │   ├── Statistics3<N>.csv       # Alignment diagnostics
-│   │   │   ├── Person<N>.csv            # Person-level output
-│   │   │   ├── BenefitUnit<N>.csv       # Benefit-unit-level output
-│   │   │   └── Household<N>.csv         # Household-level output
+│   │   │   ├── WealthIncomeStatistics.csv  # Income and wealth; Gini, S-Index
+│   │   │   ├── DemographicStatistics.csv   # Demographics by age band
+│   │   │   ├── AlignmentStatistics.csv     # Alignment diagnostics
+│   │   │   ├── LabourStatistics.csv        # Labour market transitions and participation
+│   │   │   ├── HealthStatistics.csv        # Population health by age band
+│   │   │   ├── WellbeingByGender.csv       # Wellbeing by gender, ages 25-64
+│   │   │   ├── Person.csv               # Person-level output
+│   │   │   ├── BenefitUnit.csv          # Benefit-unit-level output
+│   │   │   └── Household.csv            # Household-level output
 │   │   ├── database/                    # Run-specific persistence output
 │   │   └── input/                       # Copied run input artifacts
 │   └── logs/                       # Log files (with -f flag on multirun)