Stop container objects from claiming their children's addresses by zardus · Pull Request #730 · angr/cle

zardus · 2026-08-10T00:41:08Z

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Problem

A binary that loads on its own fails inside a container backend when it is linked at the address the container was placed at. Loading tests/x86_64/1after909.cart with its child forced to a position-dependent blob at 0:

    RAISED CLEError: 'Position-DEPENDENT object None cannot be loaded at 0x0'

The message names None, because the wrapper handed the loader a stream rather than a path, and says nothing about what is in the way. The quieter half of the same defect is that a wrapped binary does not lay out like the unwrapped one — with the main object mapped high, so the free space starts at 0:

    dynamic_load of a PIC object linked at 0: plain -> 0x0, wrapped -> 0x100000

Root cause

An outer object holds no memory: a CaRT wrapper exists only to unpack the object inside it. But it is mapped first, at its linked base of 0, where both its bounds report that base — a one-byte span — and Loader._is_range_free counts every mapped object:

if o.min_addr <= va <= o.max_addr or va <= o.min_addr < va + size:
    return False

so placement counted that byte against the object the container unpacks. _possible_addresses counted it too, which is what pushes everything the loader rebases up by a full granule. The refusal itself was raise CLEError(f"Position-DEPENDENT object {obj.binary} cannot be loaded at {base_addr:#x}"), one message covering three unrelated conditions.

Fix

find_object_containing already skips outer objects; _is_range_free and _possible_addresses now do too. The boolean check becomes _describe_range_conflict, which returns a fragment naming the object in the way or the end of the address space, and CARTFile sets the child's binary_basename to the unpacked name so the message can identify it.

--- child forced to a position-DEPENDENT blob at 0x0:
    LOADED main_object=Blob mapped_base=0x0 binary_basename='1after909.cart.unpacked'
    find_object_containing(0)=Blob
    memory.load(0, 16)=7f454c46020101000000000000000000  file head=7f454c46020101000000000000000000
--- dynamic_load of a PIC object linked at 0: plain -> 0x0, wrapped -> 0x0
--- overlapping a NamedRegion: '... cannot be loaded at 0x8000000: it would overlap mmio,
    which is mapped at [0x8000000, 0x8001000]'
--- an i386 blob at 0xfffff000: '... it is 0x13fc bytes long and would run past the end of
    the 32-bit address space'

Being outer, not being memoryless, is what takes an object out of the address space: a NamedRegion backs no memory either and still reserves its range.

Testing

tests/test_overlap.py::test_outer_object_does_not_move_rebased_objects asserts rebase(wrapped) == rebase(plain), the address itself being whatever _find_safe_rebase_addr returns; ::test_outer_object_does_not_occupy_address_space loads the container's contents at 0 to reproduce the refusal; ::test_memoryless_region_still_reserves_address_space and ::test_placement_past_the_end_of_the_address_space pin the two new messages. tests/test_cart.py::test_cart_child_is_named_after_the_wrapper and ::test_cart_layout_matches_unwrapped compare against the unwrapped tests/x86_64/1after909. Every fixture is already on angr/binaries master, so nothing here waits on a companion PR.

Validation: #730 (comment)

session: sharpen

zardus · 2026-08-10T00:41:54Z

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Validation record for head 6c31c3158c9642c8b1e8b41d40d0a3ce3283db60 against baseline 46a37333f4f59b0facf8774ee743ebc4cc074e9b, Python 3.12.13, angr/binaries master 9d9d9e0079a1f67e99046f1ebb9d4625b41152de.

Re-keyed from b2fe3d9ac3d30e89e4fec70a2fb55159e9245e1a, and this was not a rebase: the baseline is the same commit it always was, and the branch gained a second commit, 6c31c315 "Assert the wrapped layout matches, not the address it lands on". It changes one line of tests/test_overlap.py and nothing else — assert rebase(wrapped) == rebase(plain) == 0 becomes assert rebase(wrapped) == rebase(plain), because the equality is the invariant while the 0 was whichever address _find_safe_rebase_addr happened to return, and cle#765 moves that search off the null page. cle/loader.py and cle/backends/cartfile.py are byte-identical to the versions every figure below was measured on. The test rows were re-run at this head anyway.

Measured configuration: a detached worktree of cle at the revision named, this workspace's pinned Python 3.12 environment, nice -n 19, no xdist, -p no:randomly. pylint is run with the CI configuration from angr/ci-settings, ci-image/conf/pylintrc, because cle declares no [tool.pylint] table and the bare defaults score about two points lower on every file.

Focused: python -m pytest tests/test_overlap.py tests/test_cart.py tests/test_rebase.py — 15 passed in 2.6 s. tests/test_rebase.py is Place rebased objects in the free space, not one granule at a time #735's, included because this branch edits the function it covers
Full suite: python -m pytest tests — 245 passed, 9 skipped in 31.9 s. The earlier record's 231 passed is master's growth, not this branch's. The skips are the pre-existing @skip("TODO") markers in tests/test_macho_bindinghelper.py
Fails without the fix: reverting cle/loader.py and cle/backends/cartfile.py to the baseline and keeping the tests gives 5 failed, 10 passed. test_outer_object_does_not_occupy_address_space raises cle.errors.CLEError: Position-DEPENDENT object None cannot be loaded at 0x0 from Loader._map_object:1057 while the blob unpacked from the CaRT file is being placed at 0 — the defect itself. test_outer_object_does_not_move_rebased_objects fails too, which is what the second commit predicts: dropping the pinned 0 does not make that test vacuous, because the wrapped and unwrapped loaders still disagree on the baseline. test_memoryless_region_still_reserves_address_space and test_placement_past_the_end_of_the_address_space get messages that do not say what is wrong, and test_cart_child_is_named_after_the_wrapper gets None for the name
Lint: pylint per changed file, this head against the baseline — cle/loader.py 9.93 -> 9.93, cle/backends/cartfile.py, tests/test_cart.py and tests/test_overlap.py all flat at 10.00. No file regresses
Fixtures: the tests read tests/x86_64/1after909.cart, tests/x86_64/1after909, tests/i386/manysum and tests/x86_64/windows/6f289eb8c8cd826525d79b195b1cf187df509d56120427b10ea3fb1b4db1b7b5.sys.cart, all on angr/binaries master. Nothing is packed or assembled at test time, so there is no companion binaries pull request

What the second commit is about. #735 made the rebase search start above the main object, so a container mapped at 0 falls below the search range in the ordinary case. The skip in _free_gaps is still load-bearing: when the main object reaches into the top half of the address space the search starts at 0, where the container's byte sits. test_outer_object_does_not_move_rebased_objects loads 1after909 high, wrapped and unwrapped, and rebases an object into each; on the baseline the wrapped loader returns 0x100000 where the unwrapped one returns 0x0. What that test now asserts is the equality alone, so it keeps failing on the baseline while surviving a later change to where the loader chooses to place an invented object. test_cart_layout_matches_unwrapped is kept as-is since it still pins wrapped/unwrapped parity in the common configuration.

The defect first showed up in a corpus sweep, on three CaRT containers holding crash dumps in minidump format (sha256 d90692031a4d4529df8efcf1b07caf45cc5892fa1a4e655294a81b719452a0b6, ee7f0cd6d7695b3cc9a633581ec38e0f6beb60473f78bfeaaeffed977df5f553, 89323f692282267d78b84ac0cf2b59dfb678bd3c626bf4cff19d7274c102156e). Nothing about minidumps is involved; that observation was made on an older revision and the reproduction above stands on its own.

Caveats: the fix is in Loader, so it applies to every outer backend, but only the CaRT path has a test. StaticArchive and Universal2 are position independent and land on 0x400000, where a non-PIE x86-64 member is linked, so they hit the same refusal; angr/binaries has no archive or fat binary with such a member, and covering them would mean assembling one during the test, which is what an earlier version of these tests did. A position-dependent object that spans an outer object's base address can now be mapped, and find_object_containing still answers None from that base upwards; before this change such an object did not load at all.

Corpus evidence, 2026-08-28

Load-only measurement over a private corpus, so objects are cited by architecture, container format and sha256 only. Each object is loaded in its own process as angr.Project(path, auto_load_libs=False, use_sim_procedures=False) with the backend its recipe names; no analysis runs. Populations are deduplicated by object digest, because a ledger row is a run and a retry gives one object several rows. The head arm was b2fe3d9a, whose production files are byte-identical to this head's, and the master arm was cle d2ecea068794d20b1f14d90eecc1bc4bc4cfa431.

1,991 distinct objects terminate in cle/loader.py:_map_object:1057, and this change does not make any of them load. The refusal is correct; what is wrong is the message.

1,989 of them are blob firmware images under a 16-bit SLEIGH language — 6502 1,625, z80 252, 65c02 61, z180 25, x86 real mode 21, cp1600 5. The other two are singletons, an ihex and a minidump, whose recipes name no base address and which the arithmetic below therefore leaves out.

_is_range_free rejects these on its first test, va + size > 2**arch.bits, and never on an overlap. That is arithmetic rather than instrumentation: over the 1,989 blobs, taking each object's size and its recipe's base address against a 16-bit space, 1,257 are larger than 64 KB outright — bank-switched cartridge and ROM dumps of 128, 256 and 512 KB — and the remaining 732 fit in 64 KB but not at the base asked for, such as a 0xa010-byte 6502 image requested at 0x8000. Nothing is left over, so no object in the population reaches the overlap loop at all.

50 of a 50-object sample are refused on master and all 50 are still refused on the head arm, as they should be. What moves is the sentence. Master says only

Position-DEPENDENT object <path> cannot be loaded at 0x8000

naming a base address that is not by itself the problem, while this head says

Position-DEPENDENT object <name> cannot be loaded at 0x8000: it is 0xa010 bytes long and would run past the end of the 16-bit address space

Hosted CI at head 6c31c3158c9642c8b1e8b41d40d0a3ce3283db60, read live 2026-08-29T20:16Z: 18 check runs, every one success, and both legacy commit statuses green — pre-commit.ci - pr and docs/readthedocs.org:cle — for 20 terminal green checks and nothing outside success. That includes ci / Lint, ci / Typecheck, ci / Build, all eleven ci / Test shards, Test (Pyodide), Test windows-2022 and Test macos-15. The workflow run is https://github.com/angr/cle/actions/runs/33237442085, concluded success at this head. The local pre-commit run --all-files row the earlier record carried is dropped in favour of pre-commit.ci - pr at this exact head.

angr-bot · 2026-08-10T01:16:50Z

Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_730

zardus · 2026-08-26T12:31:36Z

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

This came out of assembling the angr/vibr preview snapshot (every green open PR merged together, then each component's full test suite run against the result with an angr/binaries checkout). Both PRs are green on their own.

With #765 applied, this PR's tests/test_overlap.py::test_outer_object_does_not_move_rebased_objects fails with assert 18446744071563116544 == 0: #765's _find_safe_rebase_addr tries above_image before 0, so the rebased object lands at 0xFFFFFFFF80100000. The invariant this test is about (the wrapped and plain loads rebase identically) still holds; only the literal == 0 fails, so asserting the two loads agree rather than a specific address would keep it valid under #765. Details in the comment on #765, which the preview currently excludes.

An outer object holds the objects it unpacks and backs no memory, so Backend.min_addr and Backend.max_addr both report the base it was placed at: a one-byte span. _map_object inserts it into all_objects anyway, and _is_range_free counted it, so a position-dependent child linked at that same address was refused for overlapping its own container. A CaRT file keeps its linked base of 0 and blocks a child linked at 0; StaticArchive and Universal2 are position independent and land on 0x400000, which is where a non-PIE x86-64 member is linked. find_object_containing already excludes outer objects; placement now does too, in _is_range_free and in _free_gaps, where the same byte was moving the address the loader picks for the objects it rebases and laying a wrapped binary out differently from the same binary unwrapped. The same raise covered three unrelated conditions and described only the last one, so an object that simply did not fit in the architecture's address space was reported as an overlap. Split the conditions apart in _describe_range_conflict and append the reason to the message. Name the object by binary_basename, and give the object a CaRT file unpacks the wrapper's unpacked_name, so an object loaded from a stream is no longer reported as "None". Every case is covered from a fixture that is already in angr/binaries. tests/x86_64/1after909.cart is a real container: loading its contents at 0 with the blob backend reproduces the refusal exactly as the wrapped crash dumps that first showed it do. Since _free_gaps searches upward from the main object, the placement case loads the image high, which puts the free space, and so the container's byte, below it; the wrapped binary then rebases a granule away from where the unwrapped one does. The address-space case loads tests/i386/manysum as a blob near the top of the 32-bit space. No test writes a container of its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

zardus · 2026-08-28T12:57:23Z

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Full placement report before and after this change, on tests/x86_64/1after909.cart, the same binary unwrapped at tests/x86_64/1after909, and tests/i386/manysum. No file is modified: the wrapped child is loaded through lib_opts as a position-dependent blob at base_addr=0; both copies are then loaded as a blob at 0xffffffff80000000 so the free space the loader rebases into starts at 0; manysum is loaded as an i386 blob at 0xfffff000, past the end of its address space; and a NamedRegion("mmio", 0x8000000, 0x8001000) checks that a memoryless object that is not outer still reserves its range.

Before — the child is refused for overlapping its own container, a dynamically loaded PIC object lands a granule higher when wrapped, and all three refusals share one message that names neither the obstacle nor the object:

cle at the merge base, 46a3733

cle: <cle at the merge base>/cle/__init__.py
fixtures: tests/x86_64/1after909.cart (key 02f533617364f...), tests/x86_64/1after909, tests/i386/manysum
unpacked name: 1after909.cart.unpacked
--- 1after909.cart, child forced to a position-DEPENDENT blob linked at 0x0
    RAISED CLEError: 'Position-DEPENDENT object None cannot be loaded at 0x0'
      | raise CLEError(f"Position-DEPENDENT object {obj.binary} cannot be loaded at {base_addr:#x}")
      | cle.errors.CLEError: Position-DEPENDENT object None cannot be loaded at 0x0
--- layout of the wrapped binary vs. the same binary unwrapped (default load)
    plain  : [('ELF', '0x400000', '0x603857', False), ('ExternObject', '0x700000', '0x7002e7', False)]
    wrapped: [('CARTFile', '0x0', '0x0', True), ('ELF', '0x400000', '0x603857', False), ('ExternObject', '0x700000', '0x7002e7', False)]
--- where the loader rebases into, main object mapped high (blob at 0xffffffff80000000)
    container span=[0x0, 0x0] has_memory=False is_outer=True
    main_object.min_addr plain=0xffffffff80000000 wrapped=0xffffffff80000000
    dynamic_load of a PIC object linked at 0: plain -> 0x0, wrapped -> 0x100000
--- a NamedRegion has no memory either, but is not outer, so it must still reserve its range
    region has_memory=False is_outer=False
    RAISED CLEError: 'Position-DEPENDENT object /dev/zero cannot be loaded at 0x8000000'
      | raise CLEError(f"Position-DEPENDENT object {obj.binary} cannot be loaded at {base_addr:#x}")
      | cle.errors.CLEError: Position-DEPENDENT object /dev/zero cannot be loaded at 0x8000000
--- manysum as an i386 blob based at 0xfffff000, past the end of the address space
    RAISED CLEError: 'Position-DEPENDENT object <angr/binaries>/tests/i386/manysum cannot be loaded at 0xfffff000'
      | raise CLEError(f"Position-DEPENDENT object {obj.binary} cannot be loaded at {base_addr:#x}")
      | cle.errors.CLEError: Position-DEPENDENT object <angr/binaries>/tests/i386/manysum cannot be loaded at 0xfffff000

After — the child maps at 0 and owns the address, wrapped and unwrapped layouts agree, and each refusal names its cause:

with this change, b2fe3d9

cle: <cle with this change>/cle/__init__.py
fixtures: tests/x86_64/1after909.cart (key 02f533617364f...), tests/x86_64/1after909, tests/i386/manysum
unpacked name: 1after909.cart.unpacked
--- 1after909.cart, child forced to a position-DEPENDENT blob linked at 0x0
    LOADED  main_object=Blob pic=False mapped_base=0x0
    binary_basename='1after909.cart.unpacked'
    layout=[('CARTFile', '0x0', '0x0', True), ('Blob', '0x0', '0x4faf', False)]
    find_object_containing(0)=Blob
    memory.load(0, 16)=7f454c46020101000000000000000000 file head=7f454c46020101000000000000000000
--- layout of the wrapped binary vs. the same binary unwrapped (default load)
    plain  : [('ELF', '0x400000', '0x603857', False), ('ExternObject', '0x700000', '0x7002e7', False)]
    wrapped: [('CARTFile', '0x0', '0x0', True), ('ELF', '0x400000', '0x603857', False), ('ExternObject', '0x700000', '0x7002e7', False)]
--- where the loader rebases into, main object mapped high (blob at 0xffffffff80000000)
    container span=[0x0, 0x0] has_memory=False is_outer=True
    main_object.min_addr plain=0xffffffff80000000 wrapped=0xffffffff80000000
    dynamic_load of a PIC object linked at 0: plain -> 0x0, wrapped -> 0x0
--- a NamedRegion has no memory either, but is not outer, so it must still reserve its range
    region has_memory=False is_outer=False
    RAISED CLEError: 'Position-DEPENDENT object zero cannot be loaded at 0x8000000: it would overlap mmio, which is mapped at [0x8000000, 0x8001000]'
      | raise CLEError(
      | cle.errors.CLEError: Position-DEPENDENT object zero cannot be loaded at 0x8000000: it would overlap mmio, which is mapped at [0x8000000, 0x8001000]
--- manysum as an i386 blob based at 0xfffff000, past the end of the address space
    RAISED CLEError: 'Position-DEPENDENT object manysum cannot be loaded at 0xfffff000: it is 0x13fc bytes long and would run past the end of the 32-bit address space'
      | raise CLEError(
      | cle.errors.CLEError: Position-DEPENDENT object manysum cannot be loaded at 0xfffff000: it is 0x13fc bytes long and would run past the end of the 32-bit address space

The regression asserted rebase(wrapped) == rebase(plain) == 0. The equality is the invariant -- an outer object holds no address space, so wrapping a binary must not move what the loader rebases -- while the 0 is whichever address _find_safe_rebase_addr happens to return. cle#765 changes that search to try the space above the image before the null page, so the same correct layout comes out at 0xffffffff80100000 and the pinned 0 fails. Drop it. The assertion still fails on this branch's merge base, where the container's one byte at 0 pushes the wrapped loader's answer to 0x100000 against the plain loader's 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DqcAcuGLrNJViJrpdtFYCS

zardus · 2026-08-29T06:01:06Z

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Interaction with #765, and the assertion it moved

This branch and #765 are each green alone and cannot both be in a rollup: the
mono selection has been dropping #765 for it. The failure is
tests/test_overlap.py::test_outer_object_does_not_move_rebased_objects, and it
is an assertion that pinned an address rather than a disagreement about
behaviour.

test_outer_object_does_not_move_rebased_objects asserted

assert rebase(wrapped) == rebase(plain) == 0

The equality is this branch's invariant -- an outer object holds no address
space, so wrapping a binary must not move what the loader rebases. The 0 is
whichever address _find_safe_rebase_addr happens to return, and #765 changes
that search to try the space above the image before the null page. Measured on
master + #765 + this branch, both loaders answer 0xffffffff80100000:

--- plain
    <Blob Object 1after909, maps [0xffffffff80000000:0xffffffff80004faf]>
    rebase -> 0xffffffff80100000
--- wrapped
    <CARTFile Object 1after909.cart, maps [0x0:0x0]>   outer=True  has_memory=False
    <Blob Object 1after909.cart.unpacked, maps [0xffffffff80000000:0xffffffff80004faf]>
    rebase -> 0xffffffff80100000

The invariant holds; only the literal was wrong. 6c31c31 drops it, keeping
assert rebase(wrapped) == rebase(plain). The regression still fails on this
branch's own merge base (46a3733), with the failure it was written for:

>       assert rebase(wrapped) == rebase(plain)
E       assert 1048576 == 0

What that costs, stated plainly

Once #765 lands, this test stops discriminating the _free_gaps half of this
change for this fixture. Under #765 the rebase search starts above the image, so
the container's one byte at 0 is never consulted and the outer-object skip in
_free_gaps makes no difference. Measured by removing that skip on the merged
tree and re-running:

with the outer-object skip (as merged):  plain=0xffffffff80100000 wrapped=0xffffffff80100000 equal=True
without the outer-object skip:           plain=0xffffffff80100000 wrapped=0xffffffff80100000 equal=True

That is a property of #765's placement policy, not of this test: with the search
starting above the image, an outer object at 0 can only matter in the null-page
fallback, which a 64-bit address space never reaches. The _describe_range_conflict
half of the change is unaffected and stays covered by
test_outer_object_does_not_occupy_address_space, which does not rebase.

Verification

master + #765 + #730 + #721, all three at the heads below, cle's whole suite:

257 passed, 9 skipped

Heads: #765 d160d975, #730 6c31c31 (this push), #721 d124129.

session: sharpen

zardus force-pushed the feature/fix-cle-container-overlap branch 3 times, most recently from 1288a51 to b680cde Compare August 10, 2026 23:37

zardus force-pushed the feature/fix-cle-container-overlap branch from b680cde to ab3869a Compare August 22, 2026 14:03

zardus mentioned this pull request Aug 23, 2026

Place the main binary at 0x400000 only when that space is free #776

Open

zardus mentioned this pull request Aug 26, 2026

Keep a rebased object out of the null page #765

Open

zardus force-pushed the feature/fix-cle-container-overlap branch from ab3869a to b2fe3d9 Compare August 26, 2026 22:46

zardus mentioned this pull request Aug 29, 2026

Pack and unpack words struct cannot describe #721

Open

Navigation Menu

Uh oh!

Stop container objects from claiming their children's addresses - #730

Stop container objects from claiming their children's addresses#730
zardus wants to merge 2 commits into
masterfrom
feature/fix-cle-container-overlap

zardus commented Aug 10, 2026 •

edited

Loading

Uh oh!

zardus commented Aug 10, 2026 •

edited

Loading

Uh oh!

angr-bot commented Aug 10, 2026

Uh oh!

zardus commented Aug 26, 2026

Uh oh!

zardus commented Aug 28, 2026

Uh oh!

zardus commented Aug 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

zardus commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Root cause

Fix

Testing

Uh oh!

zardus commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Corpus evidence, 2026-08-28

Uh oh!

angr-bot commented Aug 10, 2026

Uh oh!

zardus commented Aug 26, 2026

Uh oh!

zardus commented Aug 28, 2026

Uh oh!

zardus commented Aug 29, 2026

Interaction with #765, and the assertion it moved

What that costs, stated plainly

Verification

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zardus commented Aug 10, 2026 •

edited

Loading

zardus commented Aug 10, 2026 •

edited

Loading