Keep a rebased object out of the null page - #765
Conversation
bc779ad to
ad49e88
Compare
|
Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_765 |
ad49e88 to
67f1f45
Compare
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Validation record for head Focused regression, rerun at Regression. Images that still map 0, and why that is correct. On the placement sample, 14 of 198 measured still map address 0: 12 are ELFCore images whose own first segment starts at 0, and the main object's placement never goes through Headline object.
Every block remaining in an invented object is a hook node. Peak RSS falls 12%. The two changes are independently sufficient here and are not additive. A 96-byte Scale. Over the scored sample, 1,966 objects contribute 144,547 blocks decoded inside invented memory, split 922 objects / 53,507 blocks where the invented object sits below the image and 1,044 / 91,040 where it sits above:
Noise floor. CFGFast is not reproducible on every input (angr/angr#6840), so the comparison is over sorted block sets rather than counts, and the stock arm was repeated in a second fresh child wherever it was cheap. Across 508 placement, 260 control and 33 synthetic objects the two stock runs disagreed 0 times. The one control-group difference observed is shown directly to be that nondeterminism rather than an effect of either change. Full local gate. cle: 228 passed, 9 skipped. angr: 2,480 passed, 46 skipped, 2 xfailed, 260 subtests passed. angr Rust: 35 passed. Workspace checks, test-inputs and all pre-commit hooks passed with no rewrite. Lint and type checks, merge-base relative, over the changed files: COFF is excluded from the credit. #761 makes a COFF object's undefined externals into imports, so angr hooks them and stops decoding them; a hooked address is exactly what the CFGFast guard lets through. On COFF the two reach the same end by different routes and are not additive, so the 41 COFF objects among the 4,327 measured are broken out and measured again with cle#761 applied to both sides rather than counted here. Rebase. Rebased onto Cross-PR conflict, noted 2026-08-26. This came out of assembling the angr/vibr preview snapshot (every green open PR merged together, then each component's full test suite run against the result with an Combined with #730 and #721, this PR fails two of their tests:
Cause: This looks like the two detectors asserting a literal placement this PR deliberately changes rather than a defect here: #730's real invariant ( Re-keyed 2026-08-28, and what moved. The opening line above named
Corpus attribution, 2026-08-28. Objects come from material that is not public and are named by architecture, container and sha256 only. A category sweep, deduplicated by object digest, scores 460,618 distinct objects and finds a recovered block starting inside memory CLE invented on 8,663 of them, 635,997 blocks in total. Intersecting each object's own loaded-object table with its flagged addresses, 3,905 of those objects — 220,690 blocks, 34.7% of the category — have a CLE pseudo-object mapped below The shape is consistent across the population: on an 8-bit blob the extern object lands at The other 4,758 objects of the category do not show that shape: every one has a sampled flagged address, and none of them has it inside a pseudo-object mapped below Decompilation snapshots, keyed to head Re-keyed 2026-09-04, after a rebase onto The rebase was forced by the sibling version pins, not by anything in this change. angr master bumped The green
Hosted CI at the new head, read 2026-09-04T14:56Z: 18 of 18 check runs concluded |
67f1f45 to
2aafe48
Compare
|
|
||
| # The lowest page of the address space. The loader hands it out only when there is nowhere else, | ||
| # because a null or uninitialized pointer in the target reads as an address inside it. | ||
| NULL_PAGE_SIZE = 0x1000 |
There was a problem hiding this comment.
Shouldn't this be based on the actual page size instead of hardcoded?
There was a problem hiding this comment.
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS
There is no actual page size to read here, so the constant is now a floor rather than the answer. What I checked:
- No archinfo
Archdefines a page-size attribute, on any architecture — surveyed AMD64, X86, ARM, AArch64, MIPS32, PPC64, S390X, RISCV64 andArchPcode(including the 24-bitavr8:LE:24:xmega); none carries any page-related attribute. - The one page-size datum cle holds is
Loader.page_size, which defaults to0x1— "the granularity with which data is mapped into memory", i.e. target page size unknown. It carries a real page size when a core dump's mappings note supplies one (elfcore.pysetsself.loader.page_sizefrom the parsed note) or when the caller states one. It is assigned inLoader.__init__before any object is mapped, so it is available where_find_safe_rebase_addrruns.
So as of d160d97 the guard is max(self.page_size, NULL_PAGE_SIZE): one target page where the loader knows the page size, and the conventional 4 KB floor where it does not — max(1, 0x1000) leaves every existing configuration unchanged. A load stating page_size=0x2000 now keeps both pages a null pointer could land in out of the placement; test_rebase.py::test_null_guard_tracks_the_loader_page_size pins that (it places the object at 0x1000 without the change and at 0x2000 with it). Full cle suite: 243 passed, 9 skipped.
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Where the loader places the objects it invents, for a blob image based at 0x90000000. the reproducerimport logging, os
logging.getLogger("cle").setLevel(logging.CRITICAL)
import cle
BIN = os.environ["BINARIES"] # a checkout of angr/binaries
T = lambda *p: os.path.join(BIN, "tests", *p)
path = T("armel", "i2c_master_read-nucleol152re.bin")
ld = cle.Loader(path, auto_load_libs=False,
main_opts={"backend": "blob", "arch": "ARMEL", "base_addr": 0x90000000})
main = ld.main_object
print(f"main object {main.min_addr:#018x}-{main.max_addr:#018x}")
extern = ld.extern_object
print(f"extern object {extern.min_addr:#018x}-{extern.max_addr:#018x}")
print(f"find_object_containing(0x0) -> {ld.find_object_containing(0)}")
print(f"find_object_containing(0x4) -> {ld.find_object_containing(4)}")Before — the extern object is placed on the null page, so address 0 is inside a real object: cle master at
|
|
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS Resolved: the two pull requests this could not be rolled up withThis pull request has been excluded from every mono rollup because it broke two
Each fix still fails on its own branch's merge base with the error it was One consequence of this change is worth recording here rather than only on #730. #730's Verificationmaster + #765 + #730 + #721 at heads session: sharpen |
_find_safe_rebase_addr starts its search at address 0 when the architecture is narrower than 32 bits or the main image reaches into the top half of the address space, so the first object the loader places itself gets page zero. That object is normally cle##externs. Page zero is then mapped, readable and executable: a null or uninitialized pointer resolves inside a real object instead of faulting, and on 8- and 16-bit targets it answers the RST, BIOS and direct-page addresses those architectures transfer control to, which are outside the image on purpose. Search the space above the image first, then the space at or above the null page, then the null page itself, so a load with nowhere else to go still works. Only the branch that used to start at 0 changes; on every other target the first candidate is the address the function already returned. check_sparse_elf asserted that the extern and TLS objects land below 0xf800. What it guards is that they stay outside the main object's span, whose memory is one backer from 0xf800 to 0xfff00fff, so anything inside it is unreachable through Loader.memory. They now land above that span rather than below it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The two tests that pin the null page free on a narrow address space build their
architecture with archinfo.ArchPcode("z80:LE:16:default"), which raises
ArchError("pypcode not installed") when the p-code bindings are absent. pypcode
is an optional dependency of cle -- it is declared under the "pcode" extra and
in the testing group, and it has no wasm build -- so the Pyodide job, which
installs neither, failed both tests on a dependency rather than on the placement
they are about.
Guard them the way tests/test_arch_detect.py already guards its p-code
detection tests, so a runner without pypcode skips them and one with it runs
them unchanged. The other four tests in the module use no p-code architecture
and are untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review asked whether the 4 KB null guard should be based on the actual page size instead of hardcoded. The target's page size is not something the stack models on the architecture: no archinfo Arch defines a page attribute, on any architecture including ArchPcode. The one page-size datum cle holds is Loader.page_size, which defaults to 1 -- data mapped byte-granularly, target page size unknown -- and carries a real value when a core dump's mappings note supplies one or the caller states one. Size the guard as max(self.page_size, 0x1000): one target page where the loader knows the page size, and the conventional 4 KB floor where it does not. Every existing configuration is unchanged, since max(1, 0x1000) is the old constant; a load stating page_size=0x2000 now keeps both pages a null pointer could land in out of the placement. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
d160d97 to
525717e
Compare
3 participants
THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS
Problem
The objects cle invents get placed on the null page. Loading
binaries/tests/armel/i2c_master_read-nucleol152re.binas an ARMEL blob at0x90000000:Address 0 is now inside a real object, so a null or uninitialized pointer in the
target reads as a live address instead of faulting, and the loader answers
find_object_containing(0)with the extern object rather than with nothing --which is the answer every consumer uses to decide whether an address is mapped at
all.
Root cause
_find_safe_rebase_addrstarts its search at zero for a whole class of images:The reasoning is sound -- an image in the top half of the address space may have
its free space underneath it, and a 16-bit address space cannot afford to waste
any -- but "underneath the image" was implemented as "from zero", and the first
gap found is then the null page itself.
Fix
Try the space above the image first, then the space at or above one page, and take
the null page only when nothing else fits. Only the branch that used to start at 0
changes; the ordinary branch is untouched. Where the loader was told the target's
page size, the guard tracks it rather than a fixed 4 KB.
Keeping the null page free is a preference, not a constraint. An ELFCore whose
first segment starts at 0 never reaches this function, and a 16-bit image with
everything else taken still gets the null page rather than failing the load -- the
case #745 names as blocking a fix.
Testing
tests/test_rebase.py::test_image_in_the_top_half_leaves_the_null_page_freeisthe run above;
test_narrow_address_space_leaves_the_null_page_freeis the z8016-bit case;
test_null_page_is_used_when_the_address_space_has_nothing_elseasserts the fallback still happens;
test_null_guard_tracks_the_loader_page_sizecovers a non-default
page_size. The first two fail on the merge base, where theextern object is placed at 0.
Fixes #745. Validation: #765 (comment)
session: sharpen