Single-valued provenance stamp: shared entities lose sources and can be swept while still contributed

[ddeboer]

Summary

The provenance stamp is single-valued, so an entity contributed by more than one dataset records only the dataset that wrote it last. The membership sweep then deletes such a document when that dataset departs the selection, even though another still-selected dataset also contributes it.

Detail

sweep.ts declares the stamp as one string:

export const SOURCE_FIELD = 'source';

and the collection field follows: { name: SOURCE_FIELD, type: 'string', facet: true }. InPlaceRebuild stamps each dataset’s output with its own IRI:

stampDocuments(documents, { [SOURCE_FIELD]: dataset.iri.toString() })

Documents are keyed by entity IRI, so when two selected datasets both contribute the same entity, the second write overwrites the first – including its source. The document ends up attributed to whichever dataset happened to be processed last in that run.

This is not an edge case in linked-data collections. It is the normal shape for shared authorities: an AAT or Wikidata Term, a Person referenced from two collections, or – with a Dataset/Publisher root type – a publisher that published two selected datasets.

Failure

1. Datasets A and B are both selected; both contribute entity E. E.source ends up as whichever of A/B was written last – say B.
2. B leaves the selection (deregistered, or no longer matching DATASET_CRITERIA).
3. departedSources(indexed, selected) returns B, and membershipSweepFilters deletes source:=B – removing E, which A still contributes and still references.
4. A is unchanged, so on the next run PROVENANCE_FILE may skip it as unchanged, and E does not come back. Any reference to E resolves to nothing from then on.

The symmetric case is milder but also wrong: while both are selected, a facet or filter over the stamp under-reports, showing E as belonging to one dataset rather than both.

Suggested direction

Make the stamp multi-valued (string[]) and union it when a document is merged, so it records every contributing dataset. The membership sweep then deletes a document only when the departed source is the last remaining one, and removes just that source from the array otherwise. This also makes the value correct for a declared, user-facing provenance field (see the companion issue) rather than merely usually-correct. The declared field is #696; the registry-endpoint extraction that makes Publisher a realistic shared entity is #695.

Adjacent, unverified

Worth confirming separately: a selection that fails or returns empty – a registry outage, say – must abort rather than sweep. departedSources compares indexed sources against the run’s selection, so an empty selection would classify every indexed source as departed. I have not checked whether something upstream of the sweep already guards this.

[ddeboer]

Postponed. The failure here only occurs when two selected datasets contribute the same entity, and no deployment currently has datasets declaring triples about shared resources.

Also worth knowing before picking this up: a multi-valued stamp fixes attribution and deletion but leaves content last-writer-wins, so a document could list two datasets while carrying only one's values. #709 covers the whole case, with the design and the reasoning.