Skip to content

Restore exported location history - #173

Open
hamza-lzr wants to merge 1 commit into
parawanderer:mainfrom
hamza-lzr:feat/import-history-102
Open

Restore exported location history#173
hamza-lzr wants to merge 1 commit into
parawanderer:mainfrom
hamza-lzr:feat/import-history-102

Conversation

@hamza-lzr

Copy link
Copy Markdown

Fixes #102

Summary

  • add stable beacon_id identity and exact restore metadata to Android history exports
  • import Android Export History ZIPs through a validated RFC 4180 parser and one atomic Room merge
  • add the My Devices picker, result/error dialogs, latest-location refresh, and map-change notification
  • reject legacy name-only exports rather than risk attaching reports to the wrong tag

Import behavior

  • existing database rows win on (beacon_id, timestamp)
  • the first duplicate row in an archive wins
  • unknown or removed tags are skipped and counted
  • malformed rows are counted; structurally damaged archives fail before persistence
  • database failures roll back the entire import

Verification

  • ./gradlew.bat testDebugUnitTest
  • targeted managed-device history import tests (10/10)
  • ./gradlew.bat testAll
  • ./gradlew.bat testAllOnDevice (0 failures, 22 intentional skips)
  • python scripts/add_strings.py --check
  • debug APK build

The full suites above passed before the final rebase. After rebasing onto current main, string completeness and diff checks pass; a local JVM rerun could not start because this session no longer has an Android SDK path configured.

@hamza-lzr

Copy link
Copy Markdown
Author

@parawanderer

@hamza-lzr
hamza-lzr deployed to Android Build September 1, 2026 17:11 — with GitHub Actions Active
parawanderer added a commit that referenced this pull request Sep 2, 2026
Two gaps, both invisible from inside.

**Nothing said the file is agent-owned.** The header names its audience -
"rules for agents working on OpenTagViewer" - and rule 10 has one table
row obliging you to add a rule when a constraint would cost somebody an
afternoon. Neither says the file is maintained by agents, that no human
reviews it, or that keeping it current is part of the work rather than a
favour. An agent picking up work here had to infer all of that, and both
inferences are load-bearing: if nobody is filtering, the agent writing is
the only check, and if the next agent starts from this file then anything
omitted is an afternoon they pay for.

That now sits directly under the title, where it is read before anything
it governs, rather than in a rule two hundred lines down.

**And nothing warned that the register does not travel.** @parawanderer
has not read this file, nor most of docs/, most docstrings, or most
commit messages. Agents wrote them. So an agent reading them cannot tell
house style from its own predecessors' output, and assumes the former.

That showed up this week. A review of #173 was drafted in this file's
voice - argued, at length, with the reasoning as the payload - and its
reader was a first-time contributor who wanted three line numbers and a
fix. Right register, wrong reader.

The loop has nobody in it: agents write the docs, a later agent reads
them as evidence of what the maintainer wants, and writes more of the
same. Nothing in the repository contradicts that, because nothing in the
repository is written by the maintainer.

Rule 16 states the boundary. The reason it argues at length is the same
reason it says not to: the length is for an agent about to undo a
constraint, and a person reading a review is not that.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@parawanderer

Copy link
Copy Markdown
Owner

Thanks for listing what you ran locally, and for the edge case handling in the DAO and parser.

CI is approved and green: 656 tests, 0 failures. I checked the report artifact rather than the tick; your 16 new tests ran rather than being skipped.

Three findings. None were caught by CI and none could have been.

1. Peak memory during import

HistoryImporter.java:191 accumulates every row into ReadResult.rows. HistoryImportDao.java:merge then builds a HashSet<Long> of every existing timestamp per beacon, and holds both across the transaction.

Ceiling is about 70,000 rows per tag-year: ~100/day from the Find My network, plus @ubrt's local sightings throttled by LocalFixWorthKeeping to 25 m moved or 15 minutes elapsed.

1 tag, 1 year 5 tags, 1 year
row list ~20 MB ~50 MB
timestamp sets ~4 MB ~10 MB

No android:largeHeap, and Chaquopy's CPython is already resident.

Do not replace the preload with per-row EXISTS. beacon_id is not a left-most prefix of any index on LocationReport; the only index is (hash_id, beacon_id, timestamp). Every check would full-scan. One scan per beacon is correct.

Fix: intern beaconId in HistoryImporter.parseRow with a HashMap<String, String> pool. Currently allocates a fresh 36-char UUID string per row.

An index on (beacon_id, timestamp) removes the preload entirely, but needs a migration, so separate PR.

2. Crash on rotation, and no progress UI

MyDevicesListActivity.java:667 subscribes without retaining the disposable. historyImported and historyImportFailed both call MaterialAlertDialogBuilder(this).show(). Activity destroyed before completion gives BadTokenException. The destroyed activity is also retained for the duration, on top of the memory in point 1.

Duration for ~180k rows:

Stage Time
parse into objects 3-5 s
timestampsFor full scan, 5 beacons 1-3 s
SHA-256 per row 1-2 s
180k inserts, TEXT primary key plus composite index, one transaction 15-40 s

30-60 s with nothing on screen.

Fix, progress: MaterialAlertDialogBuilder.setView with a CircularProgressIndicator, non-cancellable. Already used at activity_fetch_from_icloud.xml:80. ProgressDialog is deprecated. Determinate is available for the merge phase since rows.size() is known after parsing.

Fix, rotation: disposing in onDestroy cancels a 60 s import. Hold the subscription in a ViewModel instead. AppleLoginActivity.java:191 uses ViewModelProvider with AppleLoginViewModel. Acceptable as a follow-up issue if you would rather not grow this PR.

Espresso coverage for the progress UI

ImportingHistoryFromTheDeviceListTest cannot cover this. It writes a real ZIP and runs the real importer against a real database, so the import completes in milliseconds and "in progress" is not observable.

Add a seam: AppDependencies already has replaceGeocoder, replaceICloud, replaceAuthService, replaceLogRedactor, replaceBundleBuilder, all cleared by reset(). Add an importer there so a test can install a fake that blocks on a latch.

HistoryImporter already supports this: the package-private HistoryImporter(sink, clock) constructor is what your JVM test uses. Only MyDevicesListActivity.java:670 reaches past it to construct the database-backed one inline.

Assert the indicator appears, updates, and is gone on every path including each failure. Two traps from AGENTS.md: use Eventually.check rather than a bare onView when waiting, and a GONE view still matches withId, so "gone" is matches(not(isDisplayed())) or doesNotExist(), never an expected NoMatchingViewException.

3. Unexpected exceptions reported as a damaged file

HistoryImporter.java:99:

} catch (RuntimeException error) {
    // Commons CSV reports some malformed record shapes while its iterator advances.
    throw new HistoryImportException(
            HistoryImportException.Reason.INVALID_ARCHIVE,
            "History CSV structure is invalid",
            error);
}

The Commons CSV case is real: it throws from inside Iterator.next(), so a bad record surfaces during the for (CSVRecord record : parser) loop. RuntimeException is wider. An NPE in readCsvEntry lands here too, and the distinction is gone after this line.

MyDevicesListActivity.java:721:

} else {
    title = R.string.history_import_failed_title;
    message = R.string.history_import_failed_message;
}

This is the unknown-cause branch. It should reach ErrorReportActivity, the existing "an error occurred" screen that carries the failure verbatim and links the issue template. Instead it tells the user to find a different file.

onBundleExportFailed at MyDevicesListActivity.java:555 already does this correctly, and ExportingTagsReachesTheBugPageTest covers it.

Changes:

  • Add a HistoryImportException.Reason meaning unexpected, separate from INVALID_ARCHIVE.
  • Route it from historyImportFailed's else to ErrorReportActivity.
  • Pass describe(rootOf(error)), not describe(error). Rx wraps what a map throws; the wrapper puts "RuntimeException" on the page. See the comment at MyDevicesListActivity.java:575.
  • Add an EXTRA_BODY string for the import case via scripts/add_strings.py, all ten locales.
  • Add an import equivalent of ExportingTagsReachesTheBugPageTest, asserting the page is reached for an unexpected failure and not reached for a malformed archive.

Interactively co-authored by Claude Code and @parawanderer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Import a history CSV back, so a reset does not lose it permanently

2 participants