[Shirofune-Security]

The Outlook parser for Juntei (Yamato-Security/juntei#138): PST and OST stores (ANSI and Unicode PSTs, and OSTs including the 4 KiB pages and compressed blocks of Outlook 2013 and later) and .msg / .oft files, read-only and leniently, without Outlook or MAPI.

Crates

Crate	Library	What it is
ys-outlook-parser (outlook/)	outlook	One item model for every source: pst::PstStore (store header, folders, messages and FAI) and MsgItem (.msg, .oft), each item with its well-known fields (class, sender, recipients, times, bodies, attachments, embedded messages, contact / calendar / task fields) and every property as stored
ys-outlook-parser-pst (pst/)	outlook_pst	Fork of Microsoft's outlook-pst-rs crates/pst (cfb721d, imported with its history)
ys-outlook-parser-msg (msg/)	outlook_msg	New: .msg / .oft (MS-OXMSG) on cfb 0.16
ys-outlook-parser-rtf (compressed-rtf/)	compressed_rtf	Fork of outlook-pst-rs crates/compressed-rtf

What the pst fork changes

• Outlook 2013+ OSTs (wVer 36: 4 KiB pages, 512-byte block alignment, 24-byte trailers, zlib blocks) and wVerClient 12, as in outlook-pst-ost 1.3.0 (MIT).
• The read fixes still open upstream: PRs #86, #87, #88, #89, #90, #94, #95 (MIT).
• Our own fixes: a named property's GUID was looked up 3 entries too far; OLE-object attachments were looked up in the message's sub-nodes instead of the attachment's; the store kept its root hierarchy table, which leaked every store; an embedded message's attachments could not be reached.
• Read-only: the file is never opened for writing.
• Lenient by default: each defect reading can go past is a Problem and the rest is read; Validation::Strict makes it an error.
• No input makes it panic, abort or allocate without bound (indexing_slicing, arithmetic_side_effects, unwrap_used… denied), and every walk ends: B-tree, block, heap, folder and embedded-message loop and depth limits, inflated bytes capped per walk, each heap allocation the value of one property, embedded messages that hold themselves refused.

Validation

PST/OST against libpff (pypff 20260926, used only as an external oracle), every property of every store, folder, message, recipient and attachment and every attachment's data, on 66 public PST and OST files (Apache Tika, artemis, Aspose.Email examples, DFIR Artifact Museum, Enron (ZL Technologies / EDRM), GroupDocs examples, java-libpst, libyal testdata, outlook-pst-rs, pst-extractor):

Facts	Count
The same	324,175
Read only by this crate (messages, recipients and attachments libpff fails on, and a store it cannot open)	15,846
FAI (pypff cannot list them)	8,223
Recipient cells libpff gives as empty/zero though the row says they are absent	49,077
Folder PidTagEntryId / PidTagFolderType, computed as MAPI does	2,004
Objects and embedded messages libpff gives as raw node data	480
Unexplained	0

Against upstream outlook-pst on the same files: 3,557 messages read instead of 3,326, no failing message or attachment (upstream 134 and 221), 258 embedded messages (upstream 0).

MSG/OFT against extract-msg (0.58.2, external oracle): 214 public files, 40,905 properties, 109,846 facts the same, 791 different and all explained (601 are extract-msg FILETIME errors, 35 ANSI files it decodes in the wrong code page, the rest what it leaves out or adds).

Damaged input: crafted files and seeded mutations for each crate (more than 260,000 mutated .msg files and 160,000 mutated PSTs) without a panic, abort or unbounded allocation. Independent reviews of the two readers found 6 problems in msg and 8 (plus 2 minor) in pst; all are fixed with regression tests, except a quadratic walk of sector chains inside the cfb crate, which the read budget bounds.

Tests

cargo test --workspace: 217 tests, including outlook/tests/pst_oracle.rs (the libpff comparison repeated on the fixtures against libpff's committed output), msg/tests/oracle.rs (same for extract-msg) and the crafted/mutation tests. Fixtures are openly licensed (libyal testdata CC BY 4.0; java-libpst, Apache POI, msgreader Apache-2.0; Aspose.Email examples MIT; outlook-pst-rs MIT), documented with sources and hashes in each tests/fixtures/README.md.

Review fixes

• Preserve complete LZFu and MELA RTF payloads, including embedded zero bytes and any following content. The comparison with the external Python decoder now hashes every decompressed byte rather than stopping at the first zero; the public-fixture oracle was regenerated and only four RTF hashes changed.
• Report the actual recovered MELA length when RAWSIZE understates the payload, and flag the mismatch. This preserves the existing read-to-end behavior allowed by MS-OXRTFCP 2.2.3.1.
• Convert negative fractional OLE Automation dates using the absolute fraction as the time of day, following Microsoft's DATE definition. For example, -0.5 and 0.5 both represent noon on 1899-12-30.
• Keep recovered Unicode HTML in the unified item body while preserving the damaged raw property and its diagnostics.
• Decode Unicode4k HIDs with fourteen allocation bits and thirteen block bits, retaining the layout with the heap so it remains correct outside the reader scope. Classic ANSI/Unicode layouts remain unchanged. The synthetic regression exercises every nonzero fourteen-bit allocation index, a second heap block, invalid references, and classic-layout controls. The layout was independently checked against libpff’s table reader.
• Add six regressions, verified to fail before the corresponding fixes. All 187 workspace tests, strict lint checks, formatting, documentation, the 1.99 MSRV check and the wasm32 build pass locally. Re-ran the field comparisons across 214 public MSG/OFT and 66 public PST/OST files with no unexplained differences.

Additional review fixes

• Select clean Unicode text instead of an ANSI fallback when both representations are present. HTML selection now follows the MSG reader's binary, Unicode, ANSI priority, preferring a clean representation over a damaged one. Every original property remains available.
• Preserve malformed Unicode as Value::Damaged with recovered text and original bytes, including multi-valued properties. The PST reader also retains an unmatched final byte of an unterminated Unicode string instead of dropping it; the byte is included in its read budget. UnicodeValue::trailing_byte() exposes it to low-level consumers.
• Mark odd-length UTF-16BE text as lossy even when the remaining complete code units decode successfully.
• Ignore an unset (0) PST message code page so locale, parent-message and default fallback still work.
• Skip empty attachment filenames when selecting a display name. This fixes four cases in the public corpus, including one where an empty long name hid a valid short filename.
• Follow nested recovery wrappers and retain recovered categories, task progress and attachment bytes in the corresponding item fields.
• Add 12 tests (199 workspace tests total), including synthetic end-to-end MSG tests. Workspace tests, strict clippy, formatting, documentation, MSRV 1.99 and the wasm32 build pass locally.
• Re-ran all 66 public PST/OST and 214 public MSG/OFT comparisons against the recorded independent-oracle output with no unexplained differences. Also checked subjects, plain-text and HTML bodies, attachment names and bytes in the unified model against the MSG reader across all 208 readable public files, including their embedded messages (279 messages total).

Named-property recovery and memory fixes

• A single invalid PST NAMEID record previously made stream_entry() or hash_bucket() silently return only the preceding records. The collecting methods now return an error. The new stream_entries() iterator returns each record's result separately, including a partial final record; the forensic store reader reports bad records and continues to later valid ones.
• The unified model copied a string property name for each reference. A synthetic 180 KiB MSG containing 1,500 references to one 64 KiB name aborted under a 64 MiB address-space limit before this fix. The same input now succeeds under that limit, preserving all properties and names (reported peak RSS approximately 11 MiB).
• NameId::Text now contains Arc<str> so names remain shared from the MSG reader through the unified model and across clones. Rust API note: construct it with NameId::Text(text.into()) when passing a String or string literal.
• PST string names are borrowed with the new lookup_string_bytes(), decoded once per offset and shared. Different overlapping offsets are bounded to twice the source string stream's byte length plus 4096 bytes; an excess is reported without discarding previously recovered names. Repeated references to the same offset do not consume that budget again.
• Five added tests cover malformed middle records and partial tails, borrowed-string bounds, shared-name recovery and the overlap budget. All 204 workspace tests pass, as do strict clippy, formatting, documentation, MSRV 1.99 and the wasm32 build.
• Re-ran comparisons on 66 public PST/OST and 214 public MSG/OFT files: no unexplained differences. The unified MSG field checks also pass. Checked Juntei against the local parser changes without changing its working tree: all six Outlook unit tests and seven Outlook CLI/integration tests pass.

Text-decoding and timestamp review fixes

• Preserve fractional FILETIME ticks when converting OLE Automation dates. Whole-day and epoch arithmetic now uses integers, and the stored binary fraction is scaled and rounded exactly, avoiding both epoch precision loss and a second floating-point rounding near half ticks. Negative-date conventions and range checks remain covered.
• Mark invalid or unsupported String8 decoding as Value::Damaged, retaining recovered text, original bytes and the reason. This also covers multi-valued strings and prevents a lossy ANSI value from hiding a recovered Unicode representation.
• Decode PST String8 HTML with PidTagInternetCodepage when valid, while ordinary message text retains its Windows code page. An absent, zero or negative Internet encoding falls back to the message encoding.
• Ignore negative MSG encoding/locale metadata during encoding selection rather than interpreting it as a large unsigned number. The original property remains available; a synthetic Japanese-locale regression verifies the fallback.
• Add seven tests: 211 workspace tests pass, along with strict clippy, formatting, documentation, MSRV 1.99 and wasm32 checks. An independent exact-rational reference matches 137,006 OLE-date cases with no differences, including fractional boundaries, negative dates, subnormals and range failures.
• Re-ran all 66 public PST/OST and 214 public MSG/OFT comparisons against the recorded independent-parser output with no unexplained differences. The unified MSG field comparisons and all 13 Juntei Outlook unit/CLI compatibility tests pass. The PST HTML encoding edge case is covered by a targeted synthetic unit test.

Multi-value integrity and empty-field fallback fixes

• Compare each MSG multi-valued binary/String8/Unicode element with the size in its length stream. Previously, the table supplied only an element count, so an element truncated to a complete CFB stream could be presented as undamaged. Length mismatches now report the element index and declared/actual sizes while retaining every recovered byte. The check never allocates or truncates according to an untrusted declaration, ignores binary reserved words, and safely handles sparse/out-of-range indexes.
• An empty long attachment pathname no longer hides the short pathname. An empty recipient display name no longer hides PidTagRecipientDisplayName, and an empty SMTP address no longer hides PidTagEmailAddress in the MSG convenience accessor. Every original property, including empty fields, remains available.
• Add six tests covering mismatched and matching lengths across all three variable multi-value types, sparse indexes with extreme declarations, unified-model diagnostics/recovery, and attachment/recipient fallbacks. All 217 workspace tests pass, as do strict clippy, formatting, documentation, MSRV 1.99 and wasm32 checks.
• All 66 public PST/OST and 214 public MSG/OFT comparisons against recorded independent-parser output pass with no unexplained differences. An independent olefile inspection also finds zero length mismatches across 233 multi-value elements in the public corpus. Unified MSG field checks and all 13 Juntei Outlook unit/CLI compatibility tests pass.

Not in this PR

OAB, the autocomplete stream (NK2), SRS, settings caches (RoamCache, Autodiscover), Outlook's SQLite databases, and deleted messages no longer in a folder's table: follow-up PRs for #138.

Licence

RPL-1.5. pst/ and compressed-rtf/ are changed from outlook-pst-rs (Microsoft) and outlook-pst-ost 1.3.0, whose code stays under MIT (LICENSE-MIT): RPL-1.5 AND MIT, noted in each changed file's header. msg/ and outlook/ are RPL only.

🤖 Generated with Claude Code

[Shirofune-Security]

Checked ad9cc77 and ab82862; both are correct, and 4051337 corrects the text around them.

Checks

• 4 KiB-page heap ids: a 14-bit allocation index (bits 5–18) and a 13-bit block index (bits 19–31) are what libpff's table reader and its format documentation ("64-bit 4k page table value reference") use. outlook-pst-ost 1.3.0 decodes them as 512-byte-page ids, so the bug came with the fork. Every heap is read under the reader's lock, which sets the format, so large_pages is known when each heap is built.
• Uncompressed RTF: reading to the end regardless of RAWSIZE is what MS-OXRTFCP 2.2.3.1 recommends ("SHOULD").
• Negative OLE dates: taking the fraction's absolute value matches Microsoft's DATE type documentation.
• Tests: cargo test --workspace passes all 187 tests; clippy (-D warnings) and cargo fmt --check are clean.
• libpff comparison: the crate's dump of the 66 public PST and OST files (6 of them 4 KiB-page OSTs) is byte-identical before and after these commits. None of those OSTs has a heap that reaches past the 512-byte-page layout, so the libpff numbers in the README stand.
• extract-msg comparison: I reran it on the 214 public .msg and .oft files with the code before and after these commits. Both runs give 0 unexplained differences, and the only changes are 31 RTF hashes, which moved the same way on both sides.

Fixed in 4051337

• Counts: the README and the PR description gave the extract-msg counts of an earlier run. The current counts, the same before and after these commits, are 2,399 messages, recipients and attachments, 40,905 properties, 109,846 facts the same, and 791 different, all explained.
• CHANGELOG: the Fixed section lists what was wrong in the forked crates. It keeps the RTF and 4 KiB heap id entries, now described without their tests. The README lists the heap id fix with the fork's other changes to the OST reader.
• DATE citation: MS-OAUT 2.2.25 doesn't say that a negative date's fraction is taken as its absolute value. The comment in msg/src/time.rs now cites Microsoft's DATE type documentation, which does, and gives an example.

🤖 Generated with Claude Code

[Shirofune-Security]

Checked 91a93d6. The code is correct, and ce42662 corrects the text around it.

Checks

• Tests: cargo test --workspace passes all 199 tests; clippy (-D warnings) and cargo fmt --check are clean.
• Reading Unicode to the end: a Unicode value is now read to the end of its value, which is bounded (a heap allocation or sub-node) on every path, as binary values already were.
• Code page 0: a PST message whose PidTagMessageCodepage is 0 now falls back the way a .msg file already did; the .msg reader treats 0 as absent.
• libpff comparison: the crate's dump of the 66 public PST and OST files is byte-identical before and after this commit. None of them has text that is not valid UTF-16 or a Unicode value with an odd length.
• extract-msg comparison: on the 214 public .msg and .oft files the result is unchanged: 109,846 facts the same, 791 explained, 0 unexplained.

Fixed in ce42662

• Crate docs: how the outlook crate represents damaged values, and which stored version an item's fields take, is now in the crate's documentation (outlook/src/lib.rs), where cargo doc links it to Value, rather than in the repository README. It also says that an odd last byte is left out of the recovered text, not replaced by U+FFFD.
• CHANGELOG: the Fixed section lists what was wrong in the forked crates. Of these changes, that is the last byte of an odd-length Unicode value without a terminator, which was dropped and is now kept as UnicodeValue::trailing_byte. The rest changes code that is new in 1.0.0.

Juntei
With this commit, text that is not valid UTF-16 becomes a damaged value. Juntei's Properties table showed a damaged value only as its bytes, so a subject with one bad code unit would have shown as hex alone. Yamato-Security/juntei#139 (b0ec52c) now shows the recovered value, then a note with the bytes, and updates that column's description in all 15 languages. Its Cargo.lock is at ce42662.

🤖 Generated with Claude Code

[Shirofune-Security]

Checked ede8cfa. The code is correct, and c942ef4 puts its notes where they belong.

Checks

• Tests: cargo test --workspace passes all 204 tests. Clippy (-D warnings), cargo fmt --check and cargo doc (-D warnings) are clean.
• Shared names: a string name is now shared rather than copied into every property of every item that has it. Before, one long name in a crafted store's string stream, used by many named properties, was copied into each property of each item, which could reach gigabytes. The decode budget (twice the string stream plus 4 KiB) can only be reached by names that overlap, so real stores never hit it.
• lookup_string_bytes: it checks what StringEntry::read checked (an even size, inside the stream), and lookup_string gives the same results through it.
• hash_bucket: it now fails on a bad record instead of returning the records before it. Nothing in the crates calls it, and the item reader reads the entry stream with stream_entries.
• libpff comparison: the crate's dump of the 66 public PST and OST files is byte-identical before and after this commit.
• extract-msg comparison: the dump of the 214 public .msg and .oft files is byte-identical too.
• Juntei: it builds against NameId::Text(Arc<str>) without changes, and its 553 tests pass. Add parse-file outlook: PST, OST, MSG and OFT files (part of #138) juntei#139 now uses c942ef4 (16d3370).

Fixed in c942ef4

• CHANGELOG, Changed: the section lists how the crates differ from upstream. The budget for decoding a store's names now goes with the other limits there, in the "No input makes the crates panic, abort or allocate without bound" entry. Sharing string names is the API of the new outlook crate, which NameId::Text's documentation describes.
• CHANGELOG, Fixed: the bad name-map record that hid the records after it is listed after the earlier fixes from the fork, in one sentence.
• pst/README.md: reading the entry stream record by record is now described under "Lenient reading", not at the top of "Limits". lookup_string_bytes is left to its own documentation.
• outlook crate docs: one sentence now says that string names are shared and how decoding them is bounded. NameId::Text's documentation already says how to make one.

🤖 Generated with Claude Code

[Shirofune-Security]

Checked c0f54da. The code is correct, and 5bf2cc0 fixes two small text problems.

Checks

• Tests: cargo test --workspace passes all 211 tests. Clippy (-D warnings), cargo fmt --check and cargo doc (-D warnings) are clean.
• OLE dates: I checked the integer conversion by hand: the bits of a normal or subnormal fraction, rounding half up, the cutoff past 128 bits, the fraction kept positive for dates before 1899-12-30, and overflow returning None. The epoch constant is 109,205 × 864,000,000,000. The test values agree: one f64 step is 2⁻³⁸ days at day 25,569 (3.14 ticks) and 2⁻³⁷ days at day 45,000 (6.29 ticks).
• Code pages: a negative PidTagMessageCodepage is now ignored in a .msg file, as a PST already did. A PST's ANSI HTML is decoded in PidTagInternetCodepage, as a .msg file's already was.
• Comparisons: the crate's dumps of the 66 public PST and OST files and the 214 public .msg and .oft files are byte-identical before and after this commit.
• ANSI text marked damaged: I ran Juntei with --properties over the same public files before and after this commit, once with the default code page 1252 and once with 932. The Properties table has the same 36 damaged values each time, all streams missing from damaged files, so the new marking adds none on these files.

Fixed in 5bf2cc0

• Reason text: for ANSI text that isn't valid in its code page, the reason printed the encoding with Rust's Debug formatting ("text is not valid in CodePage(932)"), and Juntei shows that reason in its Properties table. It now says "text is not valid in code page 932", and the test checks both this reason and the one for an unsupported code page.
• Crate docs: damaged ANSI text is now described in the same paragraph as damaged Unicode text: bytes not valid in the code page become U+FFFD, and text in an unsupported code page is read as Windows-1252.

Yamato-Security/juntei#139 now uses 5bf2cc0 (3bcc21c). All 553 tests pass, and its third-party notices are unchanged.

🤖 Generated with Claude Code