Relocate MIPS HI16 and LO16 from the RELA addend

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Problem

A MIPS object whose relocations are RELA does not load at all:

A mips64-hilo-rela.o (Elf64_Rela): RAISED StopIteration: <no message>
A mips64-hilo-rela.o (Elf64_Rela):   at mips.py:find_matching_lo16_relocation  |  return next(

tests/mips64/mips64-hilo-rela.o, tests/mips64/mipsn32-hilo-rela.o and tests/mips/mips-hilo-unpaired.o all end in an R_MIPS_HI16 with no following %lo, which is legal — a compiler emits one whenever the low half of the address is used elsewhere or not at all. The exception escapes cle.Loader, so the object and every relocation after the failing one are lost. Applying the relocations one at a time shows the second defect underneath: the pairs that do run build the wrong address.

E mips64-hilo-rela.o, relocations applied one at a time:
    built=['0x00/0x04->0x21020 (want 0x21020)', '0x08/0x0c->0x21030 (want 0x29030)']

Root cause

R_MIPS_HI16 always ran the SHT_REL algorithm. It found its partner with a bare next that had no default:

return next(
    reloc
    for reloc in self.owner.relocs[current_hi16_index:]
    if (self.symbol == reloc.symbol and type(reloc) is R_MIPS_LO16)
)

and recovered the addend by reading the in-place field of that partner, c_int16(self.owner.memory.unpack_word(matching_lo16_reloc_dest_addr, size=2)).value. A RELA object carries its whole addend in r_addend, so that search bought nothing — and both relocations derived from GenericAbsoluteReloc, whose value is the symbol address alone, so r_addend was simply dropped: the second pair sits 0x8000 past its section symbol and came out at 0x21030.

Fix

A shared MipsHalfwordReloc base gives both relocations a value that adds self.addend when self.is_rela, an immediate_addr that picks the low halfword of the instruction on big-endian, and an implicit_addend that reads the in-place field only for REL, where the ABI does require the %lo to follow. find_matching_lo16_relocation passes None as the default, and a REL object whose partner really is missing warns instead of aborting; an unpaired HI16 is deliberately not patched at all, so its immediate keeps what the assembler left in it.

A mips64-hilo-rela.o: text=3c0800028d0910203c0a00038d4b90303c0c0003
    built=['0x00/0x04->0x21020 (want 0x21020)', '0x08/0x0c->0x29030 (want 0x29030)']
C mips-hilo-unpaired.o: warnings=['no R_MIPS_LO16 relocation matching the R_MIPS_HI16 at 0x21010']

Testing

tests/test_mips_relocations.py::TestMipsRellocations::test_mips_hilo16_rela loads the o32-RELA, n32 and mips64 objects and asserts _computed_address(ld, BASE_ADDR + hi_offset, BASE_ADDR + lo_offset) == address for each pair; ::test_mips_hi16_without_lo16 pins the warning and the untouched immediate. These replace RELA variants the tests used to synthesise in memory, and the pre-existing REL object tests/mips/mips-hilo.o is byte-identical on both sides. Every job resolves the fixture branch through binaries-ref, so the matrix does not wait on it merging.

sync: angr/binaries#210

Validation: #722 (comment)

session: sharpen

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Validation record for head 97d49c52150a021a1d77d42f0dfd173bb74ac2d4 against baseline a4fb8003198229d33c84df6a82f749729232fd31. Commands run from the cle checkout root on CPython 3.12.13 inside the workspace shell, with angr/binaries at angr/binaries#210.

What changed since the previous head. 7b5db29b built its RELA test inputs in memory, rewriting tests/mips/mips-hilo.o's Elf32_Shdr fields with struct.pack_into and appending hand-packed Elf32_Rela entries. That was the wrong shape for this defect in particular. What the sweep below found is real RELA objects — legitimate assembler output, 38 of the 40 sampled being ELFCLASS64 — reaching a pairing search that only REL needs; a RELA container synthesised by patching a REL one is precisely the shape no toolchain emits, so the test could have passed while the real format still failed. The three synthesised variants are gone and the tests now load assembled objects from angr/binaries#210. git diff 7b5db29b 97d49c52 touches tests/test_mips_relocations.py and nothing else, so every figure in the corpus section below still describes this head's production code.

• Regression: on the baseline, test_mips_hilo16_rela (both subtests) and test_mips_hi16_without_lo16 abort inside cle.Loader with a bare StopIteration raised at cle/backends/elf/relocation/mips.py:78 in find_matching_lo16_relocation — 3 failed, 2 passed; on head all pass
• Focused: pytest tests/test_mips_relocations.py — 3 passed, 2 subtests passed
• Full suite: pytest tests/ — 242 passed, 9 skipped, 2 subtests passed. The 9 skips are the pre-existing tests/test_macho_bindinghelper.py TODOs
• Lint/type: pylint and pyright per changed file against the merge base — cle/backends/elf/relocation/mips.py 10.00 -> 10.00 and badness 0.0469 -> 0.0397, tests/test_mips_relocations.py 10.00 -> 10.00 and badness 0.0 -> 0.0. The mips.py figure is a single pre-existing pyright error on both sides; the previous record reported 0.0 -> 0.0 for it, measured against an older base and pyright, and that figure was wrong for this base
• Test inputs: check-test-inputs.py --repository cle — no binaries or assembled containers outside angr/binaries, with no allow-list exception taken. 7b5db29b would not have passed this check
• Pre-commit: pre-commit run --all-files — 22 hooks passed, 2 skipped for having no files to check, none rewrote anything
• No unintended change: every ELF under tests/mips, tests/mipsel and tests/mips64 in angr/binaries at the fixture branch — 55 files — loads to a byte-identical memory image on both revisions, except the three new fixtures, which go from StopIteration to loading

The fixtures are one assembly source built for n64, n32 and o32 by GNU as 2.46, so the same five relocations reach the loader as Elf64_Rela, Elf32_Rela and Elf32_Rel. Each object holds two matched %hi/%lo pairs against section symbols and a trailing %hi whose %lo never follows it. Loaded at 0x21000 those relocations build 0x21020, 0x29030 and 0x2d034; the middle address has bit 15 set, so its low halfword sign-extends negative and the lui has to carry, which is the case the deleted test_mips_hilo16_rela_lo16_carry covered synthetically. The o32 object exercises the REL warning path, and the test asserts the warning names that one relocation rather than merely counting warnings. Digests and the exact assembler invocations are in that pull request's own validation record.

Caveats. Only the cle suite ran: the change is confined to two relocation classes in the ELF MIPS backend, and the complete workspace gate reinstalls the shared editable environment, which other work on this machine had live workers mapped into. The editables state was recomputed and matched byte for byte before entering the shell, and no .so under repos/ or .venv/ moved inode or mtime across the run, so the suite ran against the branch without disturbing them. The fixtures are big-endian, so little-endian coverage rests on the corpus modules below rather than on a unit test. tests/test_clemory.py::test_cclemory needs a C compiler and fails outside the workspace shell; it passes inside it.

Corpus population this closes, 2026-08-28

Measured at 79f75eca7f922fdbf034ff0b1fc860508b1e05d6, which git range-diff reports identical in content to the current head 7b5db29bf7b82ace29b1d631cb6cba2fce2ef4e4.

find_matching_lo16_relocation, at cle/backends/elf/relocation/mips.py:78, raises a bare StopIteration on 3,523 distinct objects of a private corpus, all of them MIPS ELF. cle master repairs none of them. This pull request closes the whole class.

40 of those objects were run at three revisions: all 40 raise StopIteration at the effort's pin and again at cle master, and 40 of 40 load at this head.

The inputs are legitimate. Reading the section headers of the 40 directly: every one carries SHT_RELA and no SHT_REL at all, and 38 of the 40 are ELFCLASS64. Under RELA r_addend is complete, so the %hi/%lo pairing the search insists on buys nothing — while the search itself demands a same-symbol R_MIPS_LO16 positioned after the R_MIPS_HI16 in the flat owner.relocs list, which the last R_MIPS_HI16 for a symbol never has. The defect is cle's, not the toolchain's.

Example object, cited by architecture, container and digest because the corpus is not public: MIPS ELF, sha256 f4fe4b63373378a6b82e033ca7ba9e1e1799504b714144696f977f21d5aa7bf5.

Method: the corpus was scanned for every object erroring inside cle, and the objects attributed to this frame were sampled 417 units over 332 objects across 91 strata of site x backend x architecture x object kind x load options; 40 of those objects belong to this site. Each was loaded three times — at this effort's pin, at cle master, and at this pull request's head in its own worktree — with every worker printing the resolved cle.__file__ so the tree under test was verified rather than assumed.

CI state, head 97d49c52150a021a1d77d42f0dfd173bb74ac2d4, read 2026-08-28T07:05Z. All 20 checks green. GET repos/angr/cle/actions/runs?head_sha=97d49c52 answers total_count: 1, so the matrix was scheduled rather than missed. GitHub Actions run 33147012636 concluded success, with all 18 jobs reporting their own conclusion as success — ci / Build, Lint, Typecheck, the ten ci / Test shards, ci / Publish Unit Tests Results, ci / Decompiler Snapshot Testing (0), Test macos-15, Test windows-2022 and Test (Pyodide) — nothing cancelled or skipped. The webhook-driven docs/readthedocs.org:cle and pre-commit.ci - pr are green as well.

That includes the three jobs outside the ci / reusable workflow, which is the interesting part for a fixture chain: cle's ci.yml runs angr/ci-settings/actions/binaries-ref in the macOS, Windows and Pyodide jobs, so each one checked out angr/binaries#210 and ran against the new objects. The matrix is not waiting on that pull request to merge.

ci / Decompiler Snapshot Testing (0) passed. That is expected rather than reassuring: the snapshot corpus is x86-64 and this change touches only the ELF MIPS relocation classes, so it cannot exercise the path.

An earlier version of this record predicted that Test macos-15 and Test windows-2022 would be red until the fixtures merged. That was read off a checkout of ci.yml weeks behind cle master, which did not yet have the resolver step; the prediction was wrong and the two jobs are green above.

[angr-bot]

Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_722

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Full relocation report for the four MIPS objects, before and after this change. text is the first bytes of .text after loading; built is the address each %hi/%lo pair actually computes beside the one the assembler intended. Row D is the pre-existing REL object, unchanged by this PR, and row E applies the relocations of mips64-hilo-rela.o one at a time with perform_relocations=False so the failure that aborts the load does not hide the ones that precede it.

Before — the three new objects do not load, and applied one at a time the second pair builds 0x21030 where it should build 0x29030:

cle at the merge base, 46a3733

cle: <cle at the merge base>/cle/__init__.py
A mips64-hilo-rela.o (Elf64_Rela): RAISED StopIteration: <no message>
A mips64-hilo-rela.o (Elf64_Rela):   at mips.py:find_matching_lo16_relocation  |  return next(
B mipsn32-hilo-rela.o (Elf32_Rela): RAISED StopIteration: <no message>
B mipsn32-hilo-rela.o (Elf32_Rela):   at mips.py:find_matching_lo16_relocation  |  return next(
C mips-hilo-unpaired.o (Elf32_Rel, trailing HI16 with no LO16): RAISED StopIteration: <no message>
C mips-hilo-unpaired.o (Elf32_Rel, trailing HI16 with no LO16):   at mips.py:find_matching_lo16_relocation  |  return next(
D mips-hilo.o (Elf32_Rel, regression): text=3c0800023c090002210810042108102c3c0800033c0900042108101c
E mips64-hilo-rela.o, relocations applied one at a time: ['R_MIPS_HI16@0x00(addend=0)->ok', 'R_MIPS_LO16@0x04(addend=0)->ok', 'R_MIPS_HI16@0x08(addend=32768)->ok', 'R_MIPS_LO16@0x0c(addend=32768)->ok', 'R_MIPS_HI16@0x10(addend=49156)->StopIteration']
    text=3c0800028d0910203c0a00028d4b10303c0c0000
    built=['0x00/0x04->0x21020 (want 0x21020)', '0x08/0x0c->0x21030 (want 0x29030)']

After — all four load, both pairs build the intended address on every ABI, and the unpaired %hi warns instead of raising:

with this change, 97d49c5

cle: <cle with this change>/cle/__init__.py
A mips64-hilo-rela.o (Elf64_Rela): text=3c0800028d0910203c0a00038d4b90303c0c0003
    built=['0x00/0x04->0x21020 (want 0x21020)', '0x08/0x0c->0x29030 (want 0x29030)']
    warnings=[]
B mipsn32-hilo-rela.o (Elf32_Rela): text=3c0800028d0910203c0a00038d4b90303c0c0003
    built=['0x00/0x04->0x21020 (want 0x21020)', '0x08/0x0c->0x29030 (want 0x29030)']
    warnings=[]
C mips-hilo-unpaired.o (Elf32_Rel, trailing HI16 with no LO16): text=3c0800028d0910203c0a00038d4b90303c0c0001
    built=['0x00/0x04->0x21020 (want 0x21020)', '0x08/0x0c->0x29030 (want 0x29030)']
    warnings=['no R_MIPS_LO16 relocation matching the R_MIPS_HI16 at 0x21010']
D mips-hilo.o (Elf32_Rel, regression): text=3c0800023c090002210810042108102c3c0800033c0900042108101c
E mips64-hilo-rela.o, relocations applied one at a time: ['R_MIPS_HI16@0x00(addend=0)->ok', 'R_MIPS_LO16@0x04(addend=0)->ok', 'R_MIPS_HI16@0x08(addend=32768)->ok', 'R_MIPS_LO16@0x0c(addend=32768)->ok', 'R_MIPS_HI16@0x10(addend=49156)->ok']
    text=3c0800028d0910203c0a00038d4b90303c0c0003
    built=['0x00/0x04->0x21020 (want 0x21020)', '0x08/0x0c->0x29030 (want 0x29030)']

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

How much of a corpus's failure surface this removes, measured rather than
argued.

Sample. 12,000 objects drawn uniformly at random, from a seeded permutation,
out of a 624,920-object internal corpus of compiler- and vendor-produced
binaries; 11,989 were retrievable and probed, 537 of them MIPS. Rates carry 95%
Wilson intervals.

Method. Each object is loaded with the catalogue's declared load recipe and
CFGFast is taken as far as the failure under test. A 1,241-object subset — the
whole of every failure class under study plus 722 objects that already reach CFG
— is probed against master (eac0e5540516b9199dd6a91933e80dc774ea3eac) and
against this branch's head (97d49c52150a021a1d77d42f0dfd173bb74ac2d4) in one
environment, so before and after are the same objects. The comparison is keyed
on exception type and function, not on file:line, because a patch that edits
the failing file moves every line below its hunk.

Before. 8 / 11,989 = 0.067% of the sample (CI 0.034–0.132) die with a
bare StopIteration out of find_matching_lo16_relocation — the unguarded
next() over an empty generator when an R_MIPS_HI16 has no R_MIPS_LO16
after it. All 8 are big-endian 64-bit MIPS Linux kernel modules, which is the
shape that carries RELA relocations rather than REL.

After. This head clears all 8, and all 8 go on to reach CFG — the whole
class. There is no residual in it.

Control. 722 objects that already reached CFG on master are unchanged on
this head — 0 of 722 differ.

This is a small class in absolute terms, and worth saying plainly: 8 objects out
of 11,989. What it buys is that a relocation shape angr never handled stops
being an uncaught StopIteration from inside a generator, which is the kind of
failure that is expensive to diagnose from a user's traceback.

The corpus is not redistributable, so its objects are described by architecture,
format and OS rather than named; none of the 8 is byte-identical to anything
tracked in angr/binaries.

session: sharpen