[mhmdk0]

Fixes #1060.

Problem

execute_operations_in_beam shards on the FilterFiles operation, so every operation after it runs once per file. For a FileSet that is containedIn an archive, the first of those is Concatenate, and it receives a single Path instead of a list[Path]:

TypeError: 'Path' object is not iterable

This breaks mlc.ReadFromCroissant and tfds.dataset_builders.CroissantBuilder for every dataset whose FileSet is extracted from a zip or tar. Dataset.records() is not affected.

Fix

Concatenate.call now accepts either a Path or a list[Path], the same way Read.call already does.

Tests

• concatenate_test.py: new unit test covering a list of paths (sequential execution), single paths (Beam execution), and a mix of both.
• datasets_test.py: added recipes/read_binary_file_by_line.json / translations_from_zip to test_beam_hermetic_loading. The existing Beam cases (simple-parquet, simple-join) have no FileSet, so Concatenate never ran under Beamin CI. Without the fix, the new case fails with the TypeError above for both 1.0 and 1.1.

Note: the issue suggested read_from_tar.json / images_with_annotations. That record set joins two FileSets, so underBeam it fails earlier with the existing NotImplementedError: Found several FilterFiles - not handled yet. and never reaches Concatenate. translations_from_zip has a single FileSet inside a zip, so it exercises this exact path.

The reproduction script from #1060 now prints the expected 3 records with their global indices.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅