Skip to content

Latest commit

 

History

History
222 lines (179 loc) · 12 KB

ARCHITECTURE.md

File metadata and controls

222 lines (179 loc) · 12 KB

Architecture

The thesis in one paragraph

Almost everything RecoverKit needs already exists and is maintained by someone else. iLEAPP and ALEAPP parse roughly 700 mobile artifacts, and they parse live rows. PhotoRec reassembles fragmented JPEGs better than anything we would write. sqlite-dissect walks freelists and write-ahead logs. pymobiledevice3 speaks the iOS 17+ userspace tunnel. wa-crypt-tools decrypts crypt15. RecoverKit writes none of that. It is a deleted-data orchestrator: it drives those tools and adds three things none of them have, described in "What is actually new" below. Everything else in this repository is a wrapper or glue, deliberately, and should stay that way.

Flow

  device (USB)                backup / image / archive
       │                                │
       ▼                                ▼
┌────────────────────────────────────────────────────┐
│ acquire/          THIN WRAPPERS                    │  pymobiledevice3 (sub)
│                   + device state probe             │  adb (sub), adbutils, pyewf
└─────────────────────┬──────────────────────────────┘
                      ▼
          ┌─────────────────────────────────┐
          │  CASE BUNDLE (on disk)          │  manifest.json + sha256 of everything
          │  extraction/ is READ-ONLY       │
          └─────────────────┬───────────────┘
                            ▼
┌────────────────────────────────────────────────────┐
│ recover/                                           │
│   trash             ★ new                          │
│   sqlite_gaps       ★ new                          │
│   correlate         ★ new                          │
│   sqlite_freelist   ← sqlite-dissect    (wrapper)  │
│   photorec          ← photorec          (wrapper)  │
│   tsk               ← pytsk3            (wrapper)  │
│   orphan_media / thumbnails             (small)    │
└─────────────────────┬──────────────────────────────┘
                      ▼
┌────────────────────────────────────────────────────┐
│ artifacts/   modules using the iLEAPP convention   │  portable in both directions
└─────────────────────┬──────────────────────────────┘
                      ▼
          ┌─────────────────────────────────┐
          │  LEDGER (ledger.db)             │  one RecoveredItem table
          └─────────────────┬───────────────┘
                            ▼
                report/  →  self-contained report.html  ──┐
                            JSONL / CSV / timeline        │
                                                          ▼
                                                    gui/ (pywebview)

The rule that holds everything up

acquire/ is the only code that touches a device. Everything else operates exclusively on a case bundle on disk.

Three consequences follow, and they are the reason the rule is absolute:

Every recovery and parsing path becomes testable with no hardware at all, which is what lets CI run the full suite on any machine. Importing an image acquired by a completely different tool gets the entire downstream pipeline for free, with no special case. And the fake-binary test harness has exactly one seam to intercept, because every subprocess call in the project funnels through external.py.

A change that reaches for a device from recover/, artifacts/ or report/ breaks all three at once, which is why it gets rejected regardless of how well it works.

Layer boundaries

Layer May depend on May not depend on
acquire/ external, case, hashing ledger, artifacts, report
recover/ bundle paths, model acquire
artifacts/ context, model acquire, report
correlate ledger only everything upstream
report/, gui/ ledger, manifest everything else

What is actually new

Three components justify the project's existence. Everything else is plumbing around tools that already work.

Cross-source correlation — recover/correlate.py

A deleted WhatsApp message typically leaves four independent traces: nothing in the message table, its body text in the Android notification history protobuf, an orphaned image file in the media directory whose message_media row is gone, and possibly a partial cell in the msgstore.db freelist. Today that is four sources, four tools, and four disconnected outputs that a human has to reconcile by hand.

RecoverKit joins them on thread key, a time window, sender, and media hash into a single item carrying a corroborations list, and raises the confidence tier because independent sources of different method classes agree. Nobody else ships this. It is the product.

Quantified negative evidence — recover/sqlite_gaps.py

Gap analysis over row-id and primary-key sequences in sms.db, msgstore.db, Photos.sqlite and MediaStore, plus tombstone records such as PLAssetDeletedRecord. The output is of the form "1,412 messages deleted from this thread; the gaps cluster between 14:10 and 14:19 UTC on 2026-03-04" with zero content recovered.

Given that Apple force-compacts sms.db, this is frequently the only messaging result on iOS that is not empty. It costs roughly 250 lines, and because it is arithmetic over sequence numbers rather than a heuristic, it does not depend on anyone trusting a judgement call.

Unified trash sweep with expiry — recover/trash.py

One pass that covers Android in-place trashed files, vendor trash directories, MediaStore trash rows, Google Photos local_trash.db and Samsung trash.db, and on iOS the ZTRASHEDSTATE flag across both the main and Syndication photo libraries plus .Trash directories.

It renders as "37 items still fully recoverable, 12 expire in under 48 hours." Highest yield, lowest effort, byte-perfect content. The countdown is the reason someone runs the tool today instead of next month, when the answer would have been different.

Licence boundary

RecoverKit is Apache-2.0. Third-party software integrates in one of two ways, and which one is a licensing decision before it is a technical one.

Imported code runs inside the RecoverKit process. Only permissive and weak-copyleft licences qualify: MIT, BSD, Apache-2.0, LGPL.

Subprocess programs run separately, exchanging data through arguments, files and pipes. RecoverKit builds the command line, launches it, and parses the output.

Everything under the GNU GPL is subprocess-only, for two independent reasons. Importing a GPL library into a distributed work makes the combined work subject to the GPL, which Apache-2.0 cannot satisfy; running a separate program at arm's length does not. And The Sleuth Kit's core is under the IBM Public License and the Common Public License, both of which are incompatible with the GPL — so a project that went GPL and also linked TSK in-process would have an unresolvable conflict. This is why Autopsy is Apache-2.0, and RecoverKit follows the same reasoning to the same conclusion.

Practically this means pymobiledevice3, pyiosbackup, wa-crypt-tools and photorec are never in dependencies and never imported. recoverkit doctor detects them and prints the install command for the detected platform. The cost is install friction; doctor is the mitigation. The benefit is that the licensing position needs no argument.

There is a second benefit that is easy to miss: forensic practitioners need to pin and validate their own tool versions anyway. RecoverKit records the version and SHA-256 of every external binary it actually invoked in the run manifest, which is more useful than a pinned dependency would have been.

Module system

RecoverKit adopts iLEAPP's __artifacts_v2__ and @artifact_processor convention verbatim, and does not import iLEAPP as a library.

Adopting the convention means a module written here is a valid iLEAPP module and vice versa. Any of their ~700 parsers can be adopted by copying one file, with attribution, and our deleted-data modules can go upstream unchanged. Inventing a different dictionary shape would buy nothing and forfeit that permanently.

Not importing it means their internal refactors cannot break our releases. iLEAPP has no stable public API and no PyPI story worth pinning a forensic tool to. So context.py is our own roughly 80-line object exposing the surface their processors expect. Their modules run in our runner; ours run in theirs.

Their CLIs are invoked as subprocesses against the same extraction/ directory, with the resulting LAVA output ingested into the ledger as method=live_row through a small pinned adapter. Their standalone sub-libraries — ccl_segb, blackboxprotobuf, ccl_leveldb — are separate MIT packages and get imported directly.

The single addition on top of the convention: modules return the standard (data_headers, data_list, source_path) tuple, but data_headers uses RecoveredItem field names. ledger.ingest() maps by name and puts unrecognised headers into raw. No base classes, no registration decorators of our own.

Language and platform choices

Python only, floor 3.11. Every dependency in this domain is Python, so a second language would buy nothing and cost a build toolchain. There is no Rust, Go or C in this repository. The compiled dependencies (pytsk3, pyewf) are extras, so the base install stays a pure-Python wheel with no compiler requirement.

The GUI is pywebview over a local server, using the OS-native webview. The important consequence is that the GUI and the HTML report are the same frontend: the Jinja templates in report/templates/ serve both the exported self-contained report, which works from a USB stick with no server, and the desktop window, which adds endpoints for launching scans and showing progress. One frontend, two delivery modes, which is why the GUI costs roughly 400 lines instead of a parallel project.

Windows is a supported platform for both acquisition and analysis, which imposes pathlib everywhere, no string path joining, no assumption of / or of case sensitivity, and never shell=True. Anything requiring a Linux loopback mount is Linux-only and reports itself as such in doctor.

Guardrails, and what they are worth

A mandatory --authorization string, recorded verbatim in the manifest and printed in the report header. Sources opened read-only and hashed before and after. A per-run manifest capturing every input hash, output hash, external tool version and module that ran.

Stated honestly: hashing the source before and after is only meaningful for static images and files, where equality is asserted and a mismatch fails loudly. A live device mutates continuously. For those, RecoverKit records a pre and post device-state snapshot instead and states plainly that live acquisition is not bit-repeatable. Claiming otherwise would be exactly the kind of forensic overreach this tool exists to avoid.

The authorisation flag is a speed bump and an audit-trail entry, not an access control. RecoverKit deliberately does not attempt ownership detection or consent verification, because such heuristics are trivially defeated and would let the project claim a safety property it does not have. See ACCEPTABLE_USE.md.