Fix Clemory backer removal and the BackedCGC backend - #718
Conversation
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Validation record for head Re-keyed from The fixture revision on the opening line is the one this re-measurement used; the figures the earlier record carried were measured against Measured configuration: a detached worktree of cle at the revision named, this workspace's pinned Python 3.12 environment,
Removed rather than re-keyed. The pyright badness figures and the "cle and angr built from source together" ecosystem run (cle 207 passed/9 skipped, angr 2475 passed/46 skipped/2 xfailed/260 subtests, Caveats: the backend still marks dump-only regions non-executable, as it always intended to; a CGC replay that needs an executable stack passes Hosted CI at head Re-keyed 2026-09-04, after a rebase onto cle master Hosted CI at
Re-keyed 2026-09-04, after an amendment at the same base. The branch moved from That is what the per-file type gate wanted. Under the rule the hosted job applies -- The four pre-existing Every figure above still describes this head. The Hosted CI at
|
|
Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_718 |
2d584bf to
f8ea7c4
Compare
f8ea7c4 to
b104fe9
Compare
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS
Before — every removal raises, so the backend never gets past its first backer: cle masterAfter — removal finds its target, and the dump replaces every mapping except the code: with this change |
b104fe9 to
b3e60a7
Compare
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS A measurement of what this change does to a real corpus — which here bounds the Sample. 12,000 objects drawn uniformly at random, from a seeded permutation, Method. Each object is loaded with the catalogue's declared load recipe and Result. Nothing moves. All 722 control objects still reach CFG, and every That is the honest reading in both directions. The sample contains 169 CGC Correction. An earlier version of the sentence above said Loading every ELF under session: sharpen |
split_backer finds its target with backers(), which recurses into nested clemories and yields the child's own bytearrays, and then removes it with remove_backer(), which only looks at self._backers, where the child itself sits. Splitting on an address inside a child would remove the whole child and put back two slices of one of the backers it held, dropping everything else. On master remove_backer raises before that happens; with #718 applied it does not. Loader.memory is that shape, since a loaded object's memory is a nested clemory inside it. The guard meant to cover this tested the leaf backers() yielded, which is never a clemory, so it could not fire. Raise unless the backer that was found is one of this clemory's own, which is the precondition for remove_backer to remove the thing being split.
b3e60a7 to
382e6ab
Compare
Loading anything with the backedcgc backend raised "Can't find backer to remove". Clemory.remove_backer() searched with bisect_right, which always lands one index past the backer whose start matches, so no removal ever found its target and BackedCGC.__init__ died on the first file backer it tried to drop. split_backer(), and add_backer(overwrite=True) through it, were broken the same way. The backend had further defects on that path: - It compared segment vaddrs and the caller's memory_backer keys, which are linked addresses, against the object's memory, which is keyed relative to the object's base. It dropped the executable segment it meant to keep and mapped the dump at the wrong addresses. - It iterated the backer list while removing from it, so every backer following a removed one was skipped. - FakeSegment assigned to is_readable, is_writable and is_executable, which are read-only properties on Region, so any non-empty memory_backer raised AttributeError before a byte of it was mapped. - Both backers are documented as optional, but a missing memory_backer raised AttributeError, and thread_registers() returned None where Backend documents a mapping and angr's SimOS calls .items() on the result. Both are empty mappings now when the caller supplies nothing. Removing a Clemory's last backer now happens for real, so _update_min_max() restores the empty state a fresh Clemory has instead of asserting; split_backer() passes through that state on every split.
382e6ab to
eb09731
Compare
2 participants
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS
Problem
Loading anything with the
backedcgcbackend fails. Onbinaries/tests/i386/patchrex/memory_scanner, with or without a process dump:The blast radius is wider than the backend.
Clemory.remove_backer()is theprimitive under
split_backer()andadd_backer(overwrite=True), so all threeraise on a
Clemoryholding backers at 0, 10 and 20:Root cause
remove_backerbisects right:bisect.bisectisbisect_right, so for a backer that starts exactly atstartit returns the index after it. The guard on the next line thencompares
self._backers[backer_idx][0] != startagainst the following backerand raises, so no removal ever found its target -- it raises whether the address
is a backer start, inside a backer, or outside every backer.
87be7b53(2025-08-14) introduced this, and nothing caught it because master has no test
that asserts anything about
remove_backerorsplit_backer.split_backerhas not split anything since: the one path through it that would do the
splitting calls
remove_backerfirst.Behind that, the backend has its own defects. It compares linked segment
vaddrs against
self.memory._backers, whose keys are relative to the objectbase -- 0x8048000 against 0x8048000 minus 0x60006c4 -- so the comparison never
matches. It also mutates
self.memory._backerswhile iterating it, which skipsevery other entry: with only the
bisect_leftfix applied,memory_scannerkeeps one executable backer and one data backer and drops the other two. It
assigns to
Region.is_readableandis_writable, which are read-onlyproperties, and
thread_registers()returnsdict.items()whereBackenddocuments a mapping.
Fix
Use
bisect_left, and resetmin_addr/max_addr/consecutivewhen the lastbacker goes, so an emptied
Clemorymatches a fresh one. In the backend,translate every address through
AT.from_lva(...).to_rva()before it reachesself.memory, walk a copy of the backer list, giveFakeSegmenta read-onlyis_executableproperty instead of assigning toRegion's, and default bothbackers to
{}sothread_registers()returns a mapping either way.memory_scannerthen loads:Testing
tests/test_backedcgc.pyloadsmemory_scannerwith a dump and with neitherbacker;
tests/test_clemory.pycoversremove_backer,split_backerandadd_backer(overwrite=True)directly, including that removing the last backerleaves
min_addr == max_addr == 0. All five fail on the merge base withValueError: Can't find backer to remove.Validation: #718 (comment)
sync: angr/angr#6795
session: sharpen