MachO: Report a symbol in an instruction-bearing section as a function - #796
MachO: Report a symbol in an instruction-bearing section as a function#796zardus wants to merge 3 commits into
Conversation
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Validation record for head Why the old code was right and is not any more. Regressions, failing before and passing after: cle The Thumb-flag commit is measured separately: seeding the even address instead of the Thumb-tagged odd one took one fixture from 1 overlapping block to 43. Suites. cle 239 → 241 passed, zero failures either side. angr 18 failed / 2,494 passed → 1 failed / 2,512 passed; the single shared failure is Effect, 34 real-world Mach-O objects of 8 MB or less, scored by the sweep's own checker:
Cross-format, 194 public fixtures compared as address sets: 165 identical on all four sets, no ELF fixture differs on any architecture, 20 Mach-O fixtures differ (symbols 0 → 3,839, functions 11,848 → 12,039, blocks 47,481 → 47,484), and one PE fixture differs by a single function and is nondeterministic on master too across five repeats with a byte-identical block set. Relationship to angr/angr#6861. They overlap rather than compose, and on this sample #6861 is subsumed: base 537, #6861 alone 103, this pair alone 45, both together 45 and byte-identical to this pair alone on functions, blocks and every guard. Both apply cleanly to each other's heads, so the overlap is behavioural, not textual. Not addressed here. Two objects still fail to load with a pre-existing Re-keyed 2026-08-29. Re-keyed from session: sharpen Re-keyed 2026-09-04, after a rebase onto cle master Hosted CI at That test arrived in cle master The Effect and Cross-format tables above were measured on a cle master that The same staleness applies to two other artifacts. angr/angr#6984's body says Re-keyed 2026-09-04 for head Measured over every object in
The other twenty are identical on all three revisions, The Mach-O row is the repair: it moves back to what cle master produces. The six ELF rows are new duplicates and are intended. Each is a Go type-equality thunk whose name the pclntab abbreviates and the symbol table spells out -- Both clauses of the comparison are load-bearing: dropping Local suite, cle only: Hosted CI at |
|
Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_796 |
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Which Mach-O symbol-table symbols are reported as functions, before and after import cle
from cle.backends.macho.symbol import SymbolTableSymbol
ld = cle.Loader(path, auto_load_libs=False)
m = ld.main_object
for s in m.symbols:
if isinstance(s, SymbolTableSymbol):
sec = m.find_section_containing(s.rebased_addr)
print(s.name, hex(s.rebased_addr), sec.name if sec else "-", s.is_function)Before — no symbol on any Mach-O object is a function, so nothing angr/cle master (a4fb800)After — the four definitions inside an instruction-bearing section are with this changeCorrected 2026-09-04: this comment first gave the denominator as 211, the number |
5978baf to
bd840e3
Compare
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS How much of a corpus's analysis surface this changes, measured rather than Sample. 12,000 objects drawn uniformly at random, from a seeded permutation, Method. Each object is loaded with the catalogue's declared load recipe Before. Of those 156,423 Mach-O symbols, After. 86,800 of the 156,423 symbols (55.5%) are classified as functions, What that +296 is, stated precisely. CFG nodes go from 53,861 to 53,864, a Control. In a separate 1,241-object subset covering every failure class in The corpus is not redistributable, so its objects are described by architecture, session: sharpen |
bd840e3 to
aae3617
Compare
`SymbolTableSymbol.is_function` returned False for every symbol in every Mach-O object, under the comment "It is not possible to decide wether a symbol is a function or not for MachOSymbols". That was not an oversight. It arrived with the original Fraunhofer contribution in 9b28f31, next to the class's other admissions -- "the symbol type of all symbols is SymbolType.TYPE_OTHER because without docs I was unable to properly map Mach-O symbol types to CLE's notion of a symbol type", "most stock functionality from Angr and related libraries WILL NOT WORK PROPERLY on MachOSymbol" -- and on its own terms it was right: an `nlist` has no equivalent of ELF's `STT_FUNC`. `n_type` states where a symbol is defined, never what it defines, so the symbol table alone cannot answer the question. What can answer it is the section. A Mach-O states in each section's own attributes whether that section holds machine instructions, and until #762 CLE could not read that back: `MachOSection.is_executable` returned the parent segment's permissions, and ld64 marks the whole of `__TEXT` r-x, so a string literal was as executable as the code beside it. Now that a section answers for itself, a symbol defined in one that holds instructions names code, and a symbol defined anywhere else does not. The predicate is that, and the two filters it needs: * Only an `N_SECT` symbol defines anything in this object at all. An undefined, common, prebound or indirect symbol names something another object defines, an `N_ABS` symbol names a constant, and a debug entry keeps its own type in `n_type`, which `sym_type` returns unmasked, so no `N_STAB` entry reaches the section test. * The address has to lie inside the section the symbol names. `__mh_execute_header` is an `N_SECT` symbol on the first section of `__TEXT` whose value addresses the Mach-O header in front of it; every linked image has one, and without this test every one of them would hand angr the file header as a function start. The new `section` property is what both tests read, and it is bounds checked: `segment_name` and `section_name` index `sections_by_ordinal` directly and raise `IndexError` on an ordinal the object does not have.
ARM keeps the Thumb flag of a definition in `n_desc` and leaves `n_value` even. ELF states the same thing in bit 0 of `st_value`, and that is where the rest of the stack reads it -- angr's CFG decides the decoding mode of a starting point from the low bit of the address. So an image whose symbols the previous commit made visible handed the CFG even addresses for Thumb code and it decoded them as ARM. On the ARMv7 FileProtection fixture, 73 of 74 function symbols are Thumb definitions, and seeding them unmarked took the CFG from 1 overlapping block to 43 -- blocks decoded twice, in two modes, from the same bytes. Normalising the address takes that back to 1, the same as before any symbol was reported, while the recovered functions still rise from 277 to 283 and the blocks from 755 to 758.
aae3617 to
6ac4ad0
Compare
register_gopclntab_symbols skipped a pclntab entry whenever any sized function
symbol sat at its address. That was safe only while no Mach-O symbol was ever
reported as a function. Once the earlier commit in this branch reports one, the
Mach-O symbol table covers all 1,888 addresses of the Go fixture under the
platform's leading underscore -- _main.main, not main.main -- and every Go name
disappears. angr's language detector matches Go by the un-prefixed names, so a
Go Mach-O stops being detected as Go at all.
Compare the name too. A symbol covers an entry when it carries the entry's own
name, or that name with the ".abi0" suffix the Go linker gives an assembly
function in the symbol table only. An entry with no name keeps the old
address-only rule, since there is nothing to compare.
Measured over every object in angr/binaries that has a Go pclntab -- 27 of the
894 ELF, PE and Mach-O files under tests/ -- seven change. GoSymbols added:
aarch64/langdetect_go.macho 0 -> 1888, which is what cle master produces; and
{aarch64,armel,i386}/langdetect_go_go1.18.10 0 -> 2 with the matching
go1.20.14 builds 0 -> 3. The other twenty are unchanged, x86_64/starling at
6079 among them.
The six ELF ones are new duplicates and they are intended. Each is a Go
type-equality thunk whose name the pclntab abbreviates and the symbol table
spells out: "type:.eq.[...]runtime.Frame" against "type:.eq.[2]runtime.Frame",
at the same address. The two names really are different, so nothing covers the
entry under the rule above and the entry is added.
Tolerating the elision would mean guessing which spelling the symbol table
used, and would trade a lost name for a duplicate one.
Both clauses are load-bearing. Dropping the ".abi0" clause adds 115 duplicates
on langdetect_go and 119 on langdetect_go_dyn; accepting the leading underscore
keeps the Mach-O loss.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T4CH4T6AJhq6vxurNpzyxU
2 participants
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS
Problem
SymbolTableSymbol.is_functionreturnsFalseunconditionally, sobinary.symbolsnever names a function on a Mach-O object. Ontests/x86_64/fauxware.macho:In angr that silently empties the CFG's symbol-derived starting points, its
function-boundary fixup, and two protections against dropping recovered
functions.
Root cause
An
nlisthas no equivalent of ELF'sSTT_FUNC, so there was nothing obviousto test; and until #762
MachOSection.is_executablereported__TEXT'swholesale r-x, so the obvious repair would have called every string literal a
function.
Fix
Now that a section states whether it holds instructions, a symbol is a function
when it is
N_SECT, its section holds instructions, and its address liesinside that section. The containment test is load-bearing:
__mh_execute_headernames__TEXT,__textbut addresses the Mach-O header, sowithout it every linked image seeds the CFG at its own header. On
fauxware.machothat gives 4 of 14 —_authenticate,_accepted,_rejectedand
_main— while_sneaky, which lives in__data, stays out.A second commit normalises ARM's Thumb flag, which Mach-O keeps in
n_desc, tothe low address bit the rest of the codebase reads it from. On
tests/armhf/FileProtection-05.armv7.macho,_mainmoves from0x83a8to0x83a9while the one ARM-mode definition,stub helpersat0xa2f0, keepsits address.
A third commit repairs what the first one breaks in the Go pclntab loader.
register_gopclntab_symbolsskipped a pclntab entry whenever any sized functionsymbol sat at its address, which was only safe while no Mach-O symbol was ever a
function. With the first commit applied, the Mach-O symbol table covers all 1,888
addresses of
aarch64/langdetect_go.macho— but under the platform's leadingunderscore,
_main.mainrather thanmain.main— so every un-prefixed Go namedisappeared, and angr's language detector matches Go on exactly those names. The
dedupe now compares the entry's name as well as its address, accepting the
.abi0suffix the Go linker gives an assembly function in the symbol table only.Seven objects move, out of every object in
angr/binariesthat has a Go pclntab(27 of the 894 ELF, PE and Mach-O files under
tests/).langdetect_go.machogoes 0 → 1,888, back to what cle master produces. The other six each gain two
or three symbols, and that is intended: they are Go type-equality thunks whose
name the pclntab abbreviates and the symbol table spells out —
type:.eq.[...]runtime.Frameagainsttype:.eq.[2]runtime.Frame— so the twonames really are different, nothing covers the entry, and the entry is added.
The other twenty objects are unchanged,
starlingat 6,079 among them. Thefull table is in the validation record.
Testing
tests/test_macho.py::test_symbol_is_functionandtest_arm_thumb_definition_carries_the_flag_in_its_addresscover the first twocommits, on existing fixtures. Both fail on master, where the function count is 0
on each image; on this head it is 4 of 14 on
fauxware.machoand 74 of 548 onthe armv7 image.
tests/test_gopclntab.py::TestGoPclntab::test_macho_binary_supplies_the_function_symbols,which cle master gained in #808, covers the third. Master's own copy of it passes
on master and fails on the first two commits of this branch with
assert 0 == 1888; this branch's copy passes here. The two are not the sametest: this branch adds a block asserting that each of those 1,888 addresses
already carries a function symbol under the
_-prefixed or.abi0spelling,which is what the address-only dedupe was matching on. That block raises
KeyErroron master, where no Mach-O symbol is a function and the map itindexes is empty, so running this branch's file against master shows an error
rather than a pass.
Validation: #796 (comment)
session: sharpen