Shallow soils (Leptosol) display a full 120 cm profile instead of truncating at the end of soil

[DerekCaelin]

In the global (HWSD) soil-ID path, soils that are shallow by definition — e.g. Lithic Leptosol (hard rock within 20 cm) are sometimes displayed with a full 120 cm profile. At others, they correctly show a profile ending in 20cm.

Possibly related: #375

Steps To Reproduce

1. Create temporary site at: 0.14594, 35.87959
2. Tap "learn more"
3. Tap "Lithic Leptosol"
4. Review properties table
5. (see issue)

Expected behavior

Display depths ending in 20cm, as in at 8.48144, 36.35144:

Actual behavior

Display depths ending in 120cm, apparently padding out the previous layers:

Additional context

Analysis and Suggestions from DeepSeek

1. HWSD2 source data encodes the end of soil with -7

HWSD2.mdb → HWSD2_LAYERS, WRB4 = 'LPli' ("Lithic Leptosol"):

Layer	Depth	SAND	SILT	CLAY	COARSE
D1	0–20 cm	51	29	20	36
D2–D7	20–200 cm	-7	-7	-7	-7

-7 is HWSD2's "no data" sentinel — effectively "no soil in this layer." The
source represents Lithic Leptosol as ~20 cm of real soil.

2. The algorithm does not truncate on that signal

The global path has no bedrock / no-data filtering (the US path does: it
strips bedrock R horizons — us_soil.py:290).

• max_comp_depth (soil_id/utils.py:528-537) returns the last layer's
  hzdepb_r (= 200 cm for a 7-layer profile), without checking whether deeper
  layers are the -7/no-data sentinel.
• agg_data_layer (soil_id/utils.py:299-317) then caps the output at
  120 cm (final range [1, 10, 20, 50, 70, 100, 120]).

Result: a shallow soil whose HWSD2 rows still list D2–D7 is padded to the
120 cm cap instead of being truncated at 20 cm.

3. The behavior is inconsistent between points

• Kenya (0.14594, 35.87959): Lithic leptosol → 0–120 cm (bug).
• Ethiopia (8.48144, 36.35144): Lithic leptosol → 0–20 cm (correct).

Same soil name, different depth. This indicates the truncation is
data-dependent — the production DB (or the ingest step that builds
hwsd2_data) carries the deep -7-style layers for some Leptosol map units
and not others, and the algorithm faithfully echoes whichever it receives.

Likely root cause

The -7 (no-data) layers are treated as real soil throughout
list_soils_global → max_comp_depth → agg_data_layer. The end-of-soil
signal is never read, so shallow profiles get extended to the 120 cm output cap.

Suggested fix

1. In list_soils_global, drop or flag horizons whose properties are the
   HWSD2 -7 (or NULL) sentinel, so max_comp_depth reflects the true soil
   depth (Leptosol ends at ~20 cm).
2. Prefer an explicit depth-to-bedrock / "soil ends here" field (this is the
   same missing signal as the bedrock model discussion), instead of inferring
   depth from raw layer rows.
3. Investigate the ingest step that builds hwsd2_data from HWSD2 — that is
   where -7 rows should be dropped/flagged, and where the Kenya-vs-Ethiopia
   inconsistency originates.

References

• soil_id/global_soil.py — list_soils_global / horizon assembly (no no-data filtering)
• soil_id/us_soil.py:290 — US path strips bedrock R horizons (contrast)
• soil_id/utils.py:528-537 — max_comp_depth
• soil_id/utils.py:299-317 — agg_data_layer (120 cm cap)
• soil_id/db.py:183-217 — get_hwsd2_profile_data (raw layer selection)

[DerekCaelin]

Jon suprised. 120 was the classic bottom cap, thought we might start with that. The not handling -7 code for sub-bedrock seemed plausible.

[johannesparty]

Root cause found: it's a profile↔component misalignment bug, not a depth/data issue

TL;DR: the shallow-soil depth detection is correct, but list_soils_global pairs each returned soil with the wrong component's profile. The Lithic Leptosol is being shown a different, deeper component's 0–120 cm profile. Same bug family as #373 (cokey-vs-index misalignment).

Reproduction (local, current main)

Called list_soils_global(lon=35.87959, lat=0.14594) (the Kenya point) and cross-checked each soil's displayed profile depth against its true depth in hwsd2_data:

component	cokey	soilDepth	displayed max depth	true DB MAX(botdep)
Lithic leptosols	110987	20	120 ❌	20
Haplic andosols	111160	200	20 ❌	200
Eutric leptosols	111007	40	120 ❌	40
Haplic calcisols	110993	200	40 ❌	200

soilDepth and the true DB depth agree for every soil (depth detection is fine), but the displayed bottom_depth — and all the property columns (texture, sand, clay, rock_fragments, cec, ph, ec) — belong to a different component. It's a swap: the shallow leptosol gets a deep soil's profile, and a deep soil (andosol) gets the leptosol's shallow one. The deep slices carry realistic varying values (rock frag 5→7→6.7→7.6→8.3), which a 20 cm soil can't produce — proving they're another component's data.

Mechanism (soil_id/global_soil.py)

1. The per-component profile lists (hz_lyrs, snd_lyrs, cly_lyrs, txt_lyrs, rf_lyrs, cec_lyrs, ph_lyrs, ec_lyrs) are built in a loop over muhorzdata_pd.groupby("cokey", sort=False) — i.e. cokey first-appearance order.
2. They're then reordered positionally by mucomp_index = mucompdata_cond_prob.index (reordered = [[lst[i] for i in mucomp_index] for lst in lists]). mucompdata_cond_prob has been through merges + a rank/compname sort, so its index does not correspond to the cokey order the lists were built in → each soil is paired with the wrong profile.
3. soilDepth escapes the scramble because it's attached via a merge on compname, not the positional reorder — which is exactly why soilDepth stays correct while the displayed profile is wrong.

This is broader than the title suggests

Leptosols are just where it's visually obvious (a shallow soil with a deep table). Every component whose position shifts in the sort is shown the wrong profile's values; deep↔deep swaps are invisible because both cap at 120 cm. So the global "Soil Properties" table is broadly unreliable, not only for shallow soils. This also explains the Kenya-vs-Ethiopia inconsistency — the scramble depends on the local component set/ordering.

Not a data problem (original -7 hypothesis ruled out)

The hwsd2_data layer data is clean in local, staging, and prod: MAX(botdep) for any leptosol = 40, zero negative/-7 sentinel rows. Component depths in the DB are correct. The 120 cm display comes entirely from the code misassignment above, not from deep/no-data layers. (The HWSD2.mdb -7 layers described earlier are dropped at ingest.)

Suggested fix

Mirror the #373 fix: key the profile lists by cokey and emit them in mucompdata_cond_prob order via a cokey → profile lookup, instead of the positional reorder. Add a regression check that each returned soil's profile cokey matches its component cokey (or that soilDepth matches the profile's own depth). Global snapshots will shift accordingly.