Skip to content

feat: load Planktivore parquet images and localizations - #94

Merged
danellecline merged 16 commits into
mainfrom
feat/ptvr-parquet-localizations
Oct 1, 2026
Merged

danellecline merged 16 commits into
mainfrom
feat/ptvr-parquet-localizations

Conversation

@danellecline

@danellecline danellecline commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Load Planktivore parquet classifications with aidata load boxes, remapping dinov3* columns to label and score, and optionally limiting the media lookup with --section.
  • Load the parquet filename column with aidata load images: --base-path locates relative files, and the hosted URL keeps the path after the config mount. A dry run skips per-file timestamp parsing, and a reference URL that returns 301 stops the load.
  • Pin tator to 1.2.3 and ignore Dependabot updates for it, so the client that still exposes tator_openapi.Configuration stays in place.

Fixes #93

Test plan

  • poetry run pytest tests/test_tap_isiis_parquet.py tests/test_load_images_base_path.py tests/test_find_section_id.py
  • Dry-run aidata load images against a Planktivore parquet with --base-path and confirm the count returns without parsing every timestamp
  • Load a small --max-images set and confirm the hosted URL is {base}/{path after mount} and returns 200
  • aidata load boxes --section <section> only matches media in that section

danellecline and others added 16 commits September 29, 2026 09:23
Remap variable dino3 model columns to label and score so aidata load boxes can ingest ISIIS parquet classifications.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the client that still exposes tator_openapi.Configuration and ignore Dependabot updates.

Co-authored-by: Cursor <cursoragent@cursor.com>
Strip a caller-supplied directory prefix when building hosted image URLs, and reject it when images are uploaded.

Co-authored-by: Cursor <cursoragent@cursor.com>
Read png and jpg names from the filename column and join relative paths onto --base-path.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@danellecline danellecline added enhancement New feature or request dependencies Pull requests that update a dependency file python Pull requests that update python code status/needs-review Ready for human review labels Oct 1, 2026
@danellecline
danellecline merged commit 9ce46df into main Oct 1, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file enhancement New feature or request python Pull requests that update python code status/needs-review Ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for loading localizations from Planktivore parquet files

1 participant