Mach-O: guard the reads of an encrypted region and survive pickling

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Problem

Every Mach-O gets a CryptSentinel for its object memory, and an LC_ENCRYPTION_INFO/_64 command tells it which part of the image is encrypted. It overrides load, store, find and backers to refuse, which also covers read, and covers unpack and pack when the access starts inside the region. It does not cover __getitem__, __setitem__, load_null_terminated_bytes, __iter__, or a word read that starts just before the region and runs into it. Those return the bytes on disk.

On binaries/tests/armhf/FileProtection-05.arm64.macho, with its own recorded range marked encrypted:

obj.memory.load(0x4688, 4)                       EncryptedDataAccessException
obj.memory[0x4688]                               246
loader.memory[base + 0x4688]                     246
loader.memory.load_null_terminated_bytes(...)    b'\xf6W\xbd\xa9\xf4O\x01\xa9\xfd{\x02\xa9\xfd\x83'
obj.memory.load(0x3FFC, 4)  ends at 0x4000       EncryptedDataAccessException
list(iter(obj.memory))[0x4688:0x468c]            ['0xf6', '0x57', '0xbd', '0xa9']

246 is the byte on disk. angr's CFGBase reads a single byte with loader.memory[addr] and a run of bytes with loader.memory.load(...), so one analysis gets both answers about the same address. The 0x3FFC line is the mirror defect: a read ending exactly at the first encrypted byte lies outside the region and is refused anyway.

A second failure needs no encryption at all. Pickle the loader for any Mach-O and the next read raises:

loader.memory.load(base + 0x4688, 4)             AttributeError: 'CryptSentinel' object has no attribute '_is_encrypted'

Root cause

CryptSentinel overrides load, store, find and backers. Clemory.__getitem__ and Clemory.__iter__ read self._backers directly and never call backers(), and load_null_terminated_bytes is built on __getitem__. ClemoryBase.unpack and ClemoryBase.pack do call backers(), but it gets a start address and no size, so it cannot see a word read that begins before the region and ends inside it.

Clemory.__getstate__ builds its state from a fixed list of slot names. CryptSentinel is the only subclass that adds attributes, and _crypt_start, _crypt_end and _is_encrypted live in __dict__, so the round trip drops them and the first guarded read dies looking for one.

_assert_unencrypted_access tested addr in range(start, end) or (addr + size) in range(start, end) or ..., which treats the exclusive end of the access as a member of the region.

Fix

Guard __getitem__ and __setitem__ with _assert_unencrypted_access. That alone breaks membership, because Clemory.__contains__ probes __getitem__ whenever the memory is not consecutive, so rva in obj.memory and Loader.find_object_containing start raising for an encrypted address on any image with a hole in it — binaries/tests/aarch64/langdetect_go.macho is one. Whether an address is mapped is a question about the memory map, not about the bytes, so __contains__ answers it with a probe that skips the guard and discards the byte; an address inside the region that no backer maps still answers False.

Guard unpack and pack too, where the format string gives the size backers() never sees; unpack_word and pack_word are built on them. Refuse __iter__ while the image is encrypted, as find already is.

Give CryptSentinel a __getstate__/__setstate__ pair that carries its three fields, using s.get() so a pickle written before this change still loads, as an unencrypted sentinel. Replace the membership test with the half-open overlap it meant: size > 0 and addr < self._crypt_end and addr + size > self._crypt_start.

Two stay open, unchanged from master: Loader.memory_ro_view, which flattens the backers into plain bytearrays when it is built and is what angr's VEX and p-code lifters read through; and backers(), which is given an address and no size, so it hands out the whole bytearray covering the region for angr's icicle engine to slice — which is also why an access straddling the region start still reaches bytes inside it through the loader's root memory. Closing either means changing cle/memory.py.

addr in loader.memory — the root memory, not an object's — now raises for an address inside the region instead of answering. Nothing in cle or angr asks that.

Byte access on a Mach-O costs one extra Python frame, about 45 ns, and load_null_terminated_bytes pays it per byte; unpack_word and pack_word pay a little more, once per call. load, store and resident memory are unchanged, on Mach-O and on ELF alike.

Testing

tests/test_macho.py gains three tests. Two load binaries/tests/armhf/FileProtection-05.arm64.macho, which records an encryption range over [0x4000, 0x8000) with cryptid=0: one pins the reads that must keep working across a pickle round trip; the other calls set_crypt_info(1, ...) and pins the reads that must be refused, iteration and a straddling word read among them, and the reads ending at the region start that must be allowed. The third loads binaries/tests/aarch64/langdetect_go.macho, whose object memory has holes, and pins that membership and object lookup still answer for an encrypted address while an address in a hole answers False. All three fail on master, and each of the six parts of the fix is individually load-bearing.

No binary here triggers the guard for real. All 18 Mach-O files in angr/binaries that carry an encryption load command record cryptid=0, and none of the 7,784 Mach-O images in the corpora we sweep carries one at all — though that walk reads top-level file magic, and a cryptid=1 App Store binary normally lives inside an .ipa. If you have a genuinely encrypted image, that is the test worth adding on top.

#788 was our own draft over the same file, and its cle/memory.py rewrite would have closed the two paths above. It is now closed, so there is no conflict left and nothing lands second. Nothing outside cle writes through the object backers() hands out, and at that branch's head d341e928, against its own baseline a4fb8003, its snapshot cache costs 4.5x on pack_word and 5.1x on pack — cle's own relocation path — keeps 35.5 MB per backers() walk of a 37.6 MB image, and turns a write-then-read loop into a full-image copy per write, 0.0005 s to 1.00 s over 200 iterations. So the two paths above stay open, and closing either still means a check inside cle/memory.py.

Validation: #817 (comment)

session: sharpen

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Validation record for head bf7b73a4db627d414d4c70235865d85f8230fe52 against baseline 0e77ade3c39a3cee05f65051e57955675e1ac21b, with angr/binaries at 003e82a2bfa641530924055695b36cec8af483ab.

Scope: cle/backends/macho/encrypted_sentinel_backer.py +53/-6 and tests/test_macho.py +99. No other file is touched, and cle/memory.py is not touched at all. git merge-tree --write-tree against master exits 0.

Local validation. Every run from an empty directory with -p no:cacheprovider and PYTHONPATH shadowing the installed package; cle.__file__ was printed from inside each process and resolved to the intended worktree.

• pytest tests/, each arm with its own tests: master 261 passed, 9 skipped, 0 failed; head 264 passed, 9 skipped, 0 failed. The difference is the three new tests.
• All three new tests fail on master. test_encryption_guard_survives_pickling fails with AttributeError: 'CryptSentinel' object has no attribute '_is_encrypted' raised from CryptSentinel.backers; the other two fail with DID NOT RAISE EncryptedDataAccessException.
• Each of the six parts of the fix is individually load-bearing, checked by reverting one at a time at this head against an unmutated control that passes all three: without the accessor guards two tests give DID NOT RAISE; without __contains__, assert backed in memory raises EncryptedDataAccessException on the gapped file; without the half-open overlap test, memory.load(0x3FFC, 4) raises on a read that ends exactly at the first encrypted byte; without the state pair the pickling test fails as above; without the unpack/pack overrides, memory.unpack_word(0x3FFE, size=4) gives DID NOT RAISE; without the __iter__ override, iter(memory) gives DID NOT RAISE.
• CryptSentinel alone round trips at pickle protocols 0 through 5, encrypted and not, and all three fields survive. The Loader is a different matter and this change does not alter it: protocols 0 and 1 raise TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled from main_object on both arms, and 2 through 5 work here and raise the AttributeError on master. A state dict written before this change, with none of the three keys, still loads as an unencrypted sentinel and reads 246.
• ruff check . and black --check .: clean, at the versions the repository's pre-commit config pins (ruff 0.16.5, black 26.5.1). ruff format is not configured here.
• pylint 4.0.7 under CI's pylintrc, master then head, with the branch checkout on PYTHONPATH so cle resolves to the tree being linted: cle/backends/macho/encrypted_sentinel_backer.py 10.00 -> 10.00, tests/test_macho.py 9.95 -> 9.96. The only message on either arm is a pre-existing C0206 in test_macho.py.
• Typecheck gate, run against a throwaway clone because it checks out its base revision and does not restore: exit 0, encrypted_sentinel_backer.py errors 4 -> 2, tests/test_macho.py 2 -> 2. Two of master's four came from range(None, None) in the predicate this change replaces.
• check-test-inputs.py: exit 0. check-stale-pins.py --rev HEAD --base refs/remotes/gh/master: exit 0.

Performance. Min over 8 runs per arm of min-of-9 over 50,000 calls, the arms interleaved and the order reversed within each round, every written value cycling so no write is a no-op. The host carries a load average around 40 on 20 cores, so single rounds move by tens of percent in both directions; interleaving is what makes the figures below reproduce.

op	master	head
macho m[a]	0.0063	0.0085
macho m[a] = v	0.0071	0.0093
macho a in m	0.0029	0.0039
macho a in m, gapped file	0.0087	0.0124
macho load(a, 4)	0.0428	0.0423
macho store(a, 4 bytes)	0.0419	0.0420
macho unpack_word	0.0413	0.0451
macho pack_word	0.0506	0.0600
elf m[a]	0.0135	0.0132
elf a in m	0.0153	0.0155
elf load(a, 4)	0.0614	0.0615
elf unpack_word	0.0635	0.0630
elf pack_word	0.0722	0.0729
maxRSS, two starling loaders (75.18 MB of backers)	205.34	205.73

• The guarded accessors cost one extra Python frame: about 45 ns per m[a], 44 ns per m[a] = v, 20 ns per membership test, 75 ns on the gapped file. load_null_terminated_bytes pays the m[a] cost per byte.
• unpack_word costs 77 ns more per call and pack_word 189 ns, which is more than a frame: the guard sizes the format string with struct.calcsize, and pack forwards *data, which is the slower call form.
• Nothing else moves. load, store and resident memory are flat on Mach-O -- the maxRSS minima sit inside within-arm spreads of 205.34-206.57 and 205.73-206.42 -- and every ELF figure is flat, in both directions, which is the noise floor.

Corpus search for a triggering input, negative:

• angr/binaries at 003e82a2bfa641530924055695b36cec8af483ab: 27 tracked Mach-O or fat files, 18 of which carry an encryption load command, all recording cryptid=0. One is 32-bit LC_ENCRYPTION_INFO (tests/armhf/FileProtection-05.armv7.macho); the other seventeen are LC_ENCRYPTION_INFO_64.
• Every binary corpus we run decompilation sweeps over, walked in full rather than sampled: 1,788,860 files, 7,784 Mach-O images, none carrying an encryption load command at all. The walk reads top-level file magic, so a Mach-O packed inside an .ipa or another archive is not counted, and that is where a cryptid=1 App Store binary normally lives.
• The parser is stdlib-only, so it cannot inherit the behaviour under test, and it was calibrated in both directions: it finds all 18 tracked files, and it prints cryptid=1 for a scratch copy of tests/armhf/FileProtection-05.arm64.macho with that one field flipped.

Paths into the region this change does not close, measured identical on both arms. Closing either needs a check inside cle/memory.py, which this pull request does not touch:

• Loader.memory_ro_view. ClemoryReadOnlyView flattens the backers into plain bytearrays at construction, so no CryptSentinel method is on the path; __getitem__, load, unpack_word and load_null_terminated_bytes through it all return the bytes on disk. angr's VEX and p-code lifters read through this object (angr/block.py:375, :499).
• backers(), which is given a start address and no access size. next(memory.backers(addr)) for any addr below the region hands back the whole bytearray covering it, and angr slices one directly at angr/engines/icicle.py:118. It is the same reason an access through the loader's root memory that starts before the region and ends inside it still reads bytes from inside: root Clemory.load, read, unpack, store and pack reach the sentinel only through backers(), so on both arms loader.memory.unpack_word(base + 0x4686, size=4) returns 0x57f60000 for a region declared at 0x4688. On the object's own memory that read is refused at this head.

Producer/consumer audit:

• No coupled consumer pull request is required.
• Loader.find_object_containing asks rva in obj_.memory, and the __contains__ override keeps that answering for an encrypted address whether or not the object's memory is consecutive. addr in loader.memory -- the root memory, not an object's -- now raises for an address inside the region instead of answering, whether the answer was True for a mapped address or False for one in a hole. All five membership tests in cle (loader.py:452, blob.py:104, backedcgc.py:72, elf.py:1493, pe_tls.py:20) ask an object's own memory.
• EncryptedDataAccessException is not a KeyError, so a caller catching KeyError around a byte read sees it propagate, the way it already propagates out of load().
• Make loader memory reads side-effect free #788, our own draft rewriting the same file, is now closed (+102/-13 there against +53/-6 here). While it was open, git merge-tree between the two branches exited 1 with a conflict in cle/backends/macho/encrypted_sentinel_backer.py and nowhere else; against master 0e77ade3 the same command exits 0, and that is unchanged.

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Every read this change moves, on the two Mach-O files the new tests use, plus a pickle round trip of the loader for one of them left as it ships, and one route that stays open. Captured on both arms with the same script; each run prints the cle it resolved. The memory_ro_view line and the last loader.memory.unpack_word line are here because they do not move: those are the two paths this change leaves open.

Before — cle master 0e77ade3:

CLE baseline

cle: .../base788/cle/__init__.py

# FileProtection-05.arm64.macho, its own LC_ENCRYPTION_INFO_64 range rva [0x4000, 0x8000)
# marked encrypted with set_crypt_info(1, 0x4000, 0x4000)
obj.memory.load(0x4688, 4)                       EncryptedDataAccessException
obj.memory[0x4688]                               246
loader.memory[base + 0x4688]                     246
loader.memory.load_null_terminated_bytes(...)    b'\xf6W\xbd\xa9\xf4O\x01\xa9\xfd{\x02\xa9\xfd\x83'
obj.memory.load(0x3FFC, 4)  ends at 0x4000       EncryptedDataAccessException
list(iter(obj.memory))[0x4688:0x468c]            ['0xf6', '0x57', '0xbd', '0xa9']
0x4688 in obj.memory                             True
find_object_containing(base + 0x4688)            MachO
loader.memory_ro_view[base + 0x4688]             246

# langdetect_go.macho, four backers with holes between them, region [0x4000, 0x16c000)
# 0x4688 is mapped; 0x16b728 falls in the hole between the second and third backer
obj.memory.consecutive                           False
obj.memory[0x4688]                               229
0x4688 in obj.memory                             True
0x16b728 in obj.memory                           False
find_object_containing(base + 0x4688)            MachO
find_object_containing(base + 0x16b728)          NoneType

# FileProtection-05.arm64.macho as it ships, cryptid 0, after a pickle round trip of the loader
loader.memory.load(base + 0x4688, 4)             AttributeError: 'CryptSentinel' object has no attribute '_is_encrypted'

# the same file with the range declared at [0x4688, 0x5688) instead, so a word read
# starting two bytes early visibly carries bytes from inside it
obj.memory.unpack_word(0x4686, size=4)           0x57f60000
loader.memory.unpack_word(base + 0x4686, size=4) 0x57f60000
obj.memory.pack_word(0x4686, 0, size=4)          None

After — with this change:

with this change

cle: .../guard/cle/__init__.py

# FileProtection-05.arm64.macho, its own LC_ENCRYPTION_INFO_64 range rva [0x4000, 0x8000)
# marked encrypted with set_crypt_info(1, 0x4000, 0x4000)
obj.memory.load(0x4688, 4)                       EncryptedDataAccessException
obj.memory[0x4688]                               EncryptedDataAccessException
loader.memory[base + 0x4688]                     EncryptedDataAccessException
loader.memory.load_null_terminated_bytes(...)    EncryptedDataAccessException
obj.memory.load(0x3FFC, 4)  ends at 0x4000       b'\x00\x00\x00\x00'
list(iter(obj.memory))[0x4688:0x468c]            EncryptedDataAccessException
0x4688 in obj.memory                             True
find_object_containing(base + 0x4688)            MachO
loader.memory_ro_view[base + 0x4688]             246

# langdetect_go.macho, four backers with holes between them, region [0x4000, 0x16c000)
# 0x4688 is mapped; 0x16b728 falls in the hole between the second and third backer
obj.memory.consecutive                           False
obj.memory[0x4688]                               EncryptedDataAccessException
0x4688 in obj.memory                             True
0x16b728 in obj.memory                           False
find_object_containing(base + 0x4688)            MachO
find_object_containing(base + 0x16b728)          NoneType

# FileProtection-05.arm64.macho as it ships, cryptid 0, after a pickle round trip of the loader
loader.memory.load(base + 0x4688, 4)             b'\xf6W\xbd\xa9'

# the same file with the range declared at [0x4688, 0x5688) instead, so a word read
# starting two bytes early visibly carries bytes from inside it
obj.memory.unpack_word(0x4686, size=4)           EncryptedDataAccessException
loader.memory.unpack_word(base + 0x4686, size=4) 0x57f60000
obj.memory.pack_word(0x4686, 0, size=4)          EncryptedDataAccessException

246, 229 and 0x57f60000 are the plaintext on disk at those addresses. obj.memory.load(0x3FFC, 4) ends exactly at the first encrypted byte, so it is outside the region and master was wrong to refuse it. langdetect_go.macho is the file whose object memory has holes: Clemory.__contains__ only probes __getitem__ when the memory is not consecutive, so it is the one where guarding __getitem__ would otherwise have taken find_object_containing with it. The pickle line needs no encryption at all: cryptid is 0 there, and it fires on every Mach-O. The last block declares the range at 0x4688 instead, because the file's own range starts on zero bytes and a straddling read of it returns zeros either way.

[angr-bot]

Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_817