Mach-O: Read the LC_UNIXTHREAD entry point per cputype - #727
Conversation
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Validation record for head Re-keyed from Measured configuration: a detached worktree of cle at the revision named, this workspace's pinned Python 3.12 environment,
The regression test loads
An earlier version of this record covered head The old Caveats:
Hosted CI at head Re-keyed 2026-09-05 to head The one conflict was the import block of The patch itself did not move. Two things measured again at this head rather than carried over:
Correcting a line above. This record says Hosted CI is re-running at this head; the run at |
|
Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_727 |
ca4a867 to
8b1c15a
Compare
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Independent corroboration from a survey of Mach-O binaries, in case it is useful They are interesting because the failure is only reachable after a different Before #729 these died in Measured over the same twenty-one objects, each loaded in an isolated
Entry points were checked against the No action requested; the branch already does the right thing. Recording the |
8b1c15a to
aba6625
Compare
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS A corpus measurement, in case it is useful for review. A decompilation sweep over 35,578 objects recorded 48 distinct Mach-O binaries Every one has the same shape: Method: the sweep's ledger stores full tracebacks, so each failure was Two details for anyone reading failure counts off a ledger like this one:
None of the 48 come from redistribution-restricted material. session: sharpen |
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Correction to my previous comment. The denominator I gave was wrong, and it I wrote "a sweep over 35,578 objects recorded 48 distinct Mach-O binaries that Measured over what has actually been swept:
So roughly 30% of the Mach-O binaries reached so far fail to load (49/163). Of those 49, 48 are the This agrees with the load-only A/B already recorded against this corpus, which session: sharpen |
Thirteen raise sites in the Mach-O backend raised CLECompatibilityError or
CLEInvalidBinaryError with no message. A caller that catches one, and any log
that records one, gets an empty string: every rejection is indistinguishable
from every other, and from the several unrelated conditions that raise the same
type. In a sweep over 35,578 objects, 48 binaries failed this way, and the only
thing separating them from a rejection that did carry a message ("Unsupported
Mach-O file type: 8...", classifiable on sight) was that someone had written
the message.
Most of these sites already computed the diagnosis and then threw it away into
a log.error immediately above the raise, where it is lost as soon as logging is
not configured at that level, and is not attached to the exception in any case.
Move that information into the exception, and add it where it did not exist:
each message now names what was found and what was expected, so the message
alone identifies the check that rejected the file.
_load_lc_unixthread's unknown-thread-flavor raise is deliberately left alone:
open PR #727 deletes that branch rather than messaging it, making an
unrecognized flavor non-fatal, and a message there would only conflict.
The regression test loads an existing ELF fixture through the Mach-O backend,
the one path of the thirteen reachable from a fixture that already exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two rebase messages name the segment they overran, and `segment`, `address`
and `reloc_type` are all None until a SET_* opcode assigns them, so reading them
for the message is only well typed once that is established. Reject a rebase
opcode that arrives before its state is set, which is a malformed blob and
previously died on an AttributeError several lines later, and hoist the segment
end out of the loop it does not vary in.
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Full load report for Before — all three loads are refused with a message-less cle at the merge base, 46a3733After — the stock fixture yields its entry point and its five segments, and both patched copies load with no entry point rather than no object: with this change, aba6625 |
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS A second corpus measurement, on a much broader and less Mach-O-heavy sample than Sample. 12,000 objects drawn uniformly at random, from a seeded permutation, Method. Each object is loaded with the catalogue's declared load recipe and Before. 350 of the 1,204 Mach-O objects fail at load — 29.1%, which agrees After. All 3 load, and all 3 complete a full The gap against the earlier 48-object figure is corpus composition, not a A 500-object control set that already reached CFG on master is unchanged — 0 of The corpus is not redistributable, so the objects are described by architecture, session: sharpen |
aba6625 to
6da0249
Compare
Thread state flavor numbers are only unique within a cputype, but _load_lc_unixthread dispatched on the flavor alone. Flavor 1 and 6 were read as ARM_THREAD_STATE and ARM_THREAD_STATE64 whatever the cputype was, and everything else aborted the load with an empty CLECompatibilityError. An x86_64 executable stores x86_THREAD_STATE64, flavor 4, so it never loaded at all. A 32-bit x86 executable stores x86_THREAD_STATE32, flavor 1, which is the same 16 words as ARM_THREAD_STATE but keeps __eip at index 10 rather than a trailing __pc, so it loaded with __gs as its entry point. Key the thread state layouts by (cputype, flavor) and cover both x86 states. Check the state against the length the command declares and against the end of the file before unpacking it; a binary truncated inside the thread state used to come back as a bare struct.error. An LC_UNIXTHREAD that cannot be read now leaves unixthread_pc unset and lets _resolve_entry report the missing entry point, because the entry point is the only thing the command contributes and the rest of the binary is still loadable.
The backend assumed every position independent MH_EXECUTE was linked at 0x100000000 on 64 bit and 0x4000 on 32 bit. Those are ld64's defaults, not properties of the format. Go's internal linker links darwin/amd64 executables at 0x1000000, and for one of those the mapped base ended up four gigabytes above every segment, so the load aborted in Loader._map_object on `assert obj.min_addr <= obj.max_addr` before any analysis could start. Read the vmaddr of __TEXT out of the load commands instead. That is the address the mach header itself lands at and what __mh_execute_header resolves to, so it is the linked base by definition. The old constants stay as the fallback for a binary that declares no __TEXT, and every ld64-linked executable already puts __TEXT exactly where they said, so nothing changes for those. This is also what makes the LC_UNIXTHREAD change observable. Go's linker is the toolchain still emitting LC_UNIXTHREAD instead of LC_MAIN, so every binary that exercises that path is one this assumption rejected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The test assembled its own Mach-O executables with struct.pack. Test inputs belong in angr/binaries, and a hand-built container is worse than a stray binary file in the wrong repository, because it is shaped to make the test pass: this one linked __TEXT at 0x100000000, where ld64 puts it and where nothing carrying an LC_UNIXTHREAD is actually linked. The suite went green while every real binary that uses the command still failed to load. Load tests/x86_64/terramate.macho instead, the terramate executable out of the official Homebrew bottle for tenv 4.15.1. It is a Go-linked x86_64 macOS executable, so it takes its entry point from an x86_THREAD_STATE64 carried by LC_UNIXTHREAD, the flavor that used to abort the load with an empty CLECompatibilityError. The two malformed cases overwrite a single 32 bit field of that same fixture in a temp copy, which is how a bad input is made from a known good object. Two groups of cases went with the assembler: - The arm, arm64 and 32 bit x86 thread states. ld64 stopped emitting LC_UNIXTHREAD long ago and Go's linker only reaches for it on darwin/amd64, so there is no real object left that carries those states to test against. The layouts stay in the table; they are simply not covered. - "Thread state running past the end of the file", which needs LC_UNIXTHREAD to be the last thing in the file. It is the sixth of fourteen commands in a real binary, so a file truncated inside its thread state has lost every segment too and the load fails on an empty backer well before the check matters. The check stays in the parser, where it keeps a short read from surfacing as a bare struct.error, but no real container reaches it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
6da0249 to
8b59ffb
Compare
2 participants
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS
Problem
An x86_64 executable that carries its entry point in
LC_UNIXTHREADis refused outright, with an exception carrying no message.tests/x86_64/terramate.machois a Go binary, and Go's internal linker still emitsLC_UNIXTHREADrather thanLC_MAIN:The raise sits inside load-command parsing, so nothing else about the binary is parsed either — its five segments, its sections and its symbol table are all lost with the entry point.
Root cause
MachO._load_lc_unixthreadchose the thread state layout from the flavor field alone:Mach-O flavor numbers are only unique within a cputype. Flavor 4 is
x86_THREAD_STATE64, which matches neither arm, so terramate takes theelse. The same predicate reads flavor 1 on 32-bit x86 —x86_THREAD_STATE32— asARM_THREAD_STATE, which would report__gsrather than__eip; no fixture here exercises that path.Fix
The layouts are keyed on
(cputype, flavor), withCPUType,X86ThreadFlavorandARMThreadFlavorenums inmacho_enums.py, and the state is bounded by the declared word count and the end of the file. A layout cle does not know, a count too small for its flavor, or a truncated file logs a warning and leavesunixthread_pcunset rather than aborting: the entry point is all this command contributes, and_resolve_entryalready reports a binary that has none.It also reads
linked_basefrom the vmaddr of__TEXTfor a position-independentMH_EXECUTErather than the ld64 default, because Go links darwin/amd64 at0x1000000and the load would otherwise abort inLoader._map_objectonce the entry point is read correctly.Testing
tests/test_macho_unixthread.py::test_entry_point_comes_from_the_thread_stateassertsobj.entry == 0x1081180on the stock fixture;::test_flavor_without_a_known_layout_still_loadsand::test_thread_state_shorter_than_its_flavor_still_loadspatch one 32-bit field of the command in a temporary copy and pin that the rest of the binary still loads. All three fail on the merge base, where even the stock fixture is refused. The fixture is onangr/binariesmaster.Validation: #727 (comment)
session: sharpen