Scale the placement fallback to the address space

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Problem

cle.Loader cannot load an AVR gcc runtime archive, and reports the 64 KiB
address space full with a sixteenth of it in use:

  File "cle/loader.py", line 1036, in _map_object
    base_addr = self._find_safe_rebase_addr(obj_size)
  File "cle/loader.py", line 1095, in _find_safe_rebase_addr
    raise CLEOperationError("Ran out of room in address space")
cle.errors.CLEOperationError: Ran out of room in address space

The archive holds 28 members, which cle maps at 0x70 (21 of them), 0x110 (6)
and 0x1460 (1). Twenty-seven are placed; _gcov_info_to_gcda.o at 0x1460 is
not. At that point 0xf90 of 0x10000 bytes are occupied and 0xf070 is free,
spread over sixteen gaps whose largest is 0xf90.

It is not one archive. Debian Ports gcc-avr 1:14.2.0-2 ships 114 archives under
usr/lib/gcc/avr/14.2.0/, and all 57 libgcov.a among them fail this way.

Root cause

Loader._find_safe_rebase_addr tries each alignment in turn and takes the first
gap that fits:

alignments = [self._rebase_granularity]
alignments += [a for a in (0x1000, 1) if a < self._rebase_granularity]

The ladder is right; the constant in it is not. AVR's whole address space is
0x10000, so the 1 MiB granule never fits after the first object and the page
rung does all the work -- and a page is a sixteenth of the space. The first
sixteen members land at 0x0, 0x1000 ... 0xf000, one per page, each leaving
a hole of at most 0xf90. Everything after that packs into the low holes at byte
alignment, and the first member larger than a hole cannot go anywhere. The byte rung
below does not rescue this: by the time it runs, the page rung has already spent the
space.

Fix

Scale the page rung to the address space:

fallback = max(1, min(0x1000, limit >> 12))
alignments = [self._rebase_granularity]
alignments += [min(a, fallback) for a in (0x1000, 1) if a < self._rebase_granularity]

0x1000 is an absolute number in a decision about proportion. The rung above it
is already proportionate: 0x100000 is a 4096th of a 32-bit address space and
0x1000 is a 4096th of a 24-bit one, so limit >> 12 generalises the two
constants already here instead of adding a third. It is 0x1000 exactly when
limit >= 2**24, so every address space of 24 bits or wider gets the alignments
it gets today, and 12 is the largest shift with that property. Which rungs
exist is still decided by the caller's rebase_granularity exactly as before, so
a caller that asks for 0x1000 or less gets an identical ladder. On AVR the
ladder becomes [0x100000, 0x10, 1] and all 28 members fit in 0x23f0 bytes.

A caller's own rebase_granularity is deliberately not capped. Capping it also
fixes AVR, and it moves the addresses cle/tests/test_relocated.py and
angr/tests/sim/test_accuracy.py assert under rebase_granularity=0x1000000 on
i386, mips, ppc and armel, and moves 24-bit architectures as well. An explicit
granularity is a request; the rung the loader falls back to when that request
does not fit is cle's own choice, and is the part that was wrong.

Placement is first-fit, so a finer rung is a better heuristic and not a
guarantee: packing tighter can leave a differently shaped hole, and the
validation record has the measured rate both ways.

Testing

tests/test_rebase.py::test_archive_members_fill_a_16_bit_address_space loads
the archive and asserts all 28 members are placed, disjoint, inside 2**16 and
each findable through find_object_containing. On the merge base it fails with
CLEOperationError: Ran out of room in address space at cle/loader.py:1095;
with this change it passes.

Over Debian's 114 AVR archives, the 57 libgcov.a go from 0 loaded to 57. Full detail, including the placement A/B and the cases where
this packs worse, is in the validation record.

This needs the fixture in angr/binaries#228 merged first, or the new test
has no input to load.

Validation: #833 (comment)

session: sharpen

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Loading binaries/tests/avr/libgcov_avr31.a, the whole archive, before and after this change. Absolute store paths in the traceback are trimmed to cle/loader.py; nothing else is edited. The six Unknown reloc warnings are byte-identical on both arms and are a separate AVR relocation gap, not this change.

Before -- placement gives up on the twenty-eighth member with the address space almost empty:

angr/cle master 0e77ade

$ python -c 'import cle; cle.Loader("binaries/tests/avr/libgcov_avr31.a")'
Unknown reloc 24 on avr8:LE:16:default
Unknown reloc 25 on avr8:LE:16:default
Unknown reloc 18 on avr8:LE:16:default
Unknown reloc 2 on avr8:LE:16:default
Unknown reloc 3 on avr8:LE:16:default
Unknown reloc 4 on avr8:LE:16:default
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "cle/loader.py", line 187, in __init__
    self.initial_load_objects = self._internal_load(
                                ^^^^^^^^^^^^^^^^^^^^
  File "cle/loader.py", line 944, in _internal_load
    self._map_object(obj)
  File "cle/loader.py", line 1036, in _map_object
    base_addr = self._find_safe_rebase_addr(obj_size)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "cle/loader.py", line 1095, in _find_safe_rebase_addr
    raise CLEOperationError("Ran out of room in address space")
cle.errors.CLEOperationError: Ran out of room in address space

After -- all 28 members are placed, and they take 0x23f0 of 0x10000 bytes:

with this change

$ python -c '
import cle
ld = cle.Loader("binaries/tests/avr/libgcov_avr31.a")
members = [o for o in ld.all_objects if o.parent_object is ld.main_object]
for o in sorted(members, key=lambda o: o.min_addr):
    print(f"{o.binary_basename:24} 0x{o.min_addr:05x}-0x{o.max_addr:05x}  0x{o.max_addr - o.min_addr + 1:x}")
print(f"{len(members)} members placed in a 0x{2 ** ld.main_object.arch.bits:x}-byte address space")
'
Unknown reloc 24 on avr8:LE:16:default
Unknown reloc 25 on avr8:LE:16:default
Unknown reloc 18 on avr8:LE:16:default
Unknown reloc 2 on avr8:LE:16:default
Unknown reloc 3 on avr8:LE:16:default
Unknown reloc 4 on avr8:LE:16:default
_gcov_merge_add.o        0x00000-0x0010f  0x110
_gcov_merge_topn.o       0x00110-0x0021f  0x110
_gcov_merge_ior.o        0x00220-0x0032f  0x110
_gcov_merge_time_profile.o 0x00330-0x0039f  0x70
_gcov_interval_profiler.o 0x003a0-0x0040f  0x70
_gcov_interval_profiler_atomic.o 0x00410-0x0047f  0x70
_gcov_pow2_profiler.o    0x00480-0x004ef  0x70
_gcov_pow2_profiler_atomic.o 0x004f0-0x0055f  0x70
_gcov_topn_values_profiler.o 0x00560-0x005cf  0x70
_gcov_topn_values_profiler_atomic.o 0x005d0-0x0063f  0x70
_gcov_average_profiler.o 0x00640-0x006af  0x70
_gcov_average_profiler_atomic.o 0x006b0-0x0071f  0x70
_gcov_ior_profiler.o     0x00720-0x0078f  0x70
_gcov_ior_profiler_atomic.o 0x00790-0x007ff  0x70
_gcov_indirect_call_profiler_v4.o 0x00800-0x0086f  0x70
_gcov_time_profiler.o    0x00870-0x008df  0x70
_gcov_dump.o             0x008e0-0x009ef  0x110
_gcov_fork.o             0x009f0-0x00a5f  0x70
_gcov_execl.o            0x00a60-0x00acf  0x70
_gcov_execlp.o           0x00ad0-0x00b3f  0x70
_gcov_execle.o           0x00b40-0x00baf  0x70
_gcov_execv.o            0x00bb0-0x00c1f  0x70
_gcov_execvp.o           0x00c20-0x00c8f  0x70
_gcov_execve.o           0x00c90-0x00cff  0x70
_gcov_reset.o            0x00d00-0x00e0f  0x110
_gcov_lock_unlock.o      0x00e10-0x00e7f  0x70
_gcov.o                  0x00e80-0x00f8f  0x110
_gcov_info_to_gcda.o     0x00f90-0x023ef  0x1460
28 members placed in a 0x10000-byte address space

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Validation record for head bf20802ccc2f4b6202459cfcc539a159f397ebfa against baseline 0e77ade3c39a3cee05f65051e57955675e1ac21b.

• Regression: pytest tests/test_rebase.py -- 1 failed, 3 passed on the baseline (CLEOperationError: Ran out of room in address space at cle/loader.py:1095), 4 passed on the head
• Full suite: pytest tests/ -- 262 passed, 9 skipped
• Lint/type: cle's own pre-commit hooks ran on the commit -- ruff, black, pyupgrade and the rest, all passed
• Workspace gate: ./feature.sh test avr-granularity, 2026-09-08. Every suite that exercises this change passed: cle 262 passed 9 skipped; angr 2,917 passed, 47 skipped, 2 xfailed, 286 subtests passed; angr's Rust suite; the mono pipeline, 139 tests; pysoot; test-inputs; test-packages; and every configured pre-commit hook on both checkouts. The gate's overall return code is 1 for two failures in the workspace repository's own test_no_venv_survives checks -- AGENTS.md and angr-maintain-mono/scripts/rollup.sh still mention .venv-native, and one allowlist entry matches no line. Neither touches cle, both are in a repository this change does not modify, and both reproduce on other candidates.
• Gate coverage, quoted from the gate itself: SUITES SKIPPED (feature has not adopted, did NOT run): archinfo pypcode pyvex angr-management. A gate that skipped a suite is green over less than it appears to be; this feature adopted angr, cle and binaries, which are the repositories the change and its fixture touch.
• File digests: cle/loader.py 7d0b60d27513bdc7b7e20b4adbd42c546773fd7f2ff596fcfe0af9e86d5d69e6, tests/test_rebase.py 301c8025591daf6e8d8944b82d0aa7c2cae45cfb1b502d3ee0cbcb8e6e84055d
• Fixture: angr/binaries tests/avr/libgcov_avr31.a, 63ffbb28b90916b17ceb446eea8fef3c5523c4f719c5642324fbc72762f27e0f, 31,660 bytes

How the A/B was run

Two source trees extracted with git archive, one at the baseline and one at the head, each prepended to sys.path in its own process. Each arm's process prints the sha256 of the loader.py it imported: baseline 6da4affa360628907787b9bc227eab8b9536decad29b9b7db560ea7ab09d0695, head 7d0b60d27513bdc7b7e20b4adbd42c546773fd7f2ff596fcfe0af9e86d5d69e6. A control arm with no cle in it imports the environment's copy and the probe refuses, so the prepend is load-bearing rather than assumed. Every figure below was produced against those two digests.

Set	Baseline	Head	Layouts changed
57 Debian AVR libgcov.a, 30-34 KB	0 loaded	57 loaded	57
57 Debian AVR libgcc.a, 1.7-2.0 MB	0 loaded	0 loaded	n/a, neither arm loads one
252 angr/binaries objects, 16 architectures	252 loaded	252 loaded	1
125 angr/binaries loads, auto_load_libs=True, 500 objects	125 loaded	125 loaded	0
187 p-code languages, unaligned blob	187 loaded	187 loaded	37

• The one object that moves in the 252 is tests/avr/isqrt_atmega128.o, the only 16-bit file in the set: its kernel object goes from 0x1000-0x8fff to 0x200-0x81ff. The other 251, at 32 and 64 bits, are identical name for name and address for address. Architectures covered: AARCH64, AMD64, ARMCortexM, ARMEL, ARMHF, MIPS32, MIPS64, MIPSN32, PPC32, PPC64, RISCV64, S390X, X86, avr8, m68k, SuperH4.
• All 37 p-code languages that move are under 24 bits; none of the 150 at 24 bits or more moves. That is the arithmetic claim measured: min(0x1000, limit >> 12) is 0x1000 exactly when limit >= 2**24. The blob must not be page-aligned to see this -- with a 4096-byte blob the gap after the main object is already aligned for both ladders and 0 of 187 move.
• The libgcc.a set fails on both arms because 1.7 to 2.0 MB of members do not fit in 64 KiB. Members placed before the error: the head reaches more on 20 of 57, fewer on 14, the same on 23. On an archive that cannot fit, how far placement gets before the error is not a property to depend on.
• Debian Ports gcc-avr 1:14.2.0-2 ships exactly 114 archives under usr/lib/gcc/avr/14.2.0/, counted from the package: 57 libgcc.a and 57 libgcov.a.

Where this packs worse than master

The change is not a strict improvement, and this is the measurement that shows it.

At 16 bits with rebase_granularity left at cle's default 0x100000, driving the unmodified _find_safe_rebase_addr/_free_gaps with sizes [0x3000, 0x1000, 0xfff, 0x1000, 0x3000, 0x2000, 0xfff, 0x1001, 0xfff, 0xfff, 0xfff]: the baseline places 11 of 11, the head places 10 and raises CLEOperationError. The baseline's 0x1000 rung puts the 9th and 10th objects at 0xE000 and 0xF000, leaving the whole 0xD001-0xDFFF hole -- exactly 0xfff bytes -- for the 11th. The head's 0x10 rung fits them at 0xD010 and 0xE010, and the largest hole left is 4,081 bytes against an object needing 4,095.

Randomised search, same seed on both arms so both see byte-identical trials, 5 to 80 objects per trial, five size families (log-uniform in two ranges, uniform, resampled from the 1,596 real AVR libgcov.a member sizes, and one adversarial family drawn only from page-boundary sizes {0xfff, 0x1000, 0x1001, 0x2000, 0x2fff, 0x3000}):

Region	Trials	Head places fewer	Head places more	Identical
bits < 24 and granularity > 0x1000	60,000	104	12,712	47,184
bits >= 24 or granularity <= 0x1000	90,000	0	0	90,000

• All 104 are in the adversarial page-boundary family at 16 bits, 2.6% of its trials. In the four families not chosen with the mechanism in mind, including the real AVR member sizes, it is 0 of 48,000.
• No trial anywhere placed 2 fewer.
• The null control is checked at address level in a second sweep with a different seed: 0 layout differences in 33,000 trials at 24, 25, 32 and 64 bits or at granularity 0x1000, against 8,429 of 9,000 in region. The instrument can see a difference where one exists and reports none where the ladder is provably identical.
• In the regime this change exists for -- 150 to 420 members drawn from the real AVR size population, in a 64 KiB space -- the head placed more in 600 of 600 trials and fewer in none, turning 116 of 600 exhaustions into complete placements.

Timing

CPU time (time.process_time) around the placement work only, arms interleaved with the order flipped on alternate rounds, n = 60 per arm except where noted, medians with the full spread, load average recorded at both ends. The workload is the real member size sequence of a Debian AVR libgcc.a, which is the hardest case the ladder sees.

Input	Baseline median	Head median	Head/baseline
369 real sizes at 64 bits (identity control, three runs; the middle one is n = 24)	9.366 / 9.271 / 9.461 ms	9.521 / 9.392 / 9.369 ms	1.017, 1.013, 0.990
369 real sizes at 16 bits, the full archive (two runs; the second is n = 24)	10.118 / 10.158 ms	10.652 / 10.827 ms	1.053, 1.066
250 real sizes at 16 bits, avrtiny/double64	4.932 ms	4.258 ms	0.863
250 real sizes at 16 bits, avr51	5.061 ms	3.848 ms	0.760

• The identity control is an input where the ladder is provably unchanged and both arms place all 369 objects at the same addresses. Its three ratios straddle 1.0, so the measurement floor is about 2% and there is no evidence of a constant per-call cost for computing fallback.
• No end-to-end ratio is quoted. Placement is about 10 ms of a 15.7 s archive load, 0.07%; with n = 6 and the load average moving from 65 to 96 during the run, the smallest end-to-end difference detectable was about 15%, and a 64-bit identity control -- where nothing can change -- still measured 0.869.

The alternative that was rejected

Capping the caller's rebase_granularity itself, rather than only the fallback rung, over the same sets:

	This change	Cap the caller's granularity
57 AVR libgcov.a	57 load	57 load
The 11-object regression case above	10 of 11	10 of 11, identically
187 p-code languages	37 move, all 16-bit	51 move: the same 37 plus 14 at 24 bits
cle/tests/test_relocated.py	passes	fails: [0x8048000, 0x9000000, 0xA000000] becomes [0x8048000, 0x8100000, 0x8300000]
angr/tests/sim/test_accuracy.py fixtures	unchanged	mips, ppc and armel libc.so.6 all move, and the addresses the test asserts land on no object

It buys nothing this change does not, regresses identically on the adversarial case, and changes more.

Neighbours

git merge-tree against the heads of angr/cle #765 525717e6, #717 5b0405bb and #776 3443309f conflicts in tests/test_rebase.py only, at the append point all four branches share; cle/loader.py merges cleanly with all three, including #765's rewrite of this very loop. #730 6c31c315, which edits _free_gaps in the loop this change iterates, and #736 bde21bd5 do not conflict at all. angr/cle #735, which introduced this ladder, is merged and is an ancestor of the baseline.

Caveats

The 114 AVR archives come from the Debian Ports build of gcc-avr 1:14.2.0-2, gcc-avr_14.2.0-2_hurd-i386.deb, sha256 d1a5db88c0fc04b518ec0c70e50d85644e357a957b33205e9b18e75f34796342. Take it from Debian Ports, not Debian main: main's amd64 build of the same version ships a copy of this archive that is also 31,660 bytes but differs in 230 ar header bytes, in no member body byte, and so hashes differently.

session: sharpen

[angr-bot]

Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_833