PE: a section the file does not contain is skipped from the image but still counts toward max_addr

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

PE._get_memory_mapped_image skips a section whose raw data begins past the end of the file, and logs that it is doing so, while _register_sections registers that same section, so it still counts toward max_addr. The object ends up declaring a range nothing backs.

On the committed fixture tests/i386/windows/a94bbeed0ef51db3d3964bb0cc2cbed0adab0e47997d88f34daa92faa1a91e8a, ld.main_object.contains_addr(ld.main_object.max_addr) is True while ld.find_object_containing of that same address is None, describe_addr calls it "not part of a loaded object", and ld.memory.load(ld.main_object.sections_map[".rdata"].vaddr, 1) raises KeyError.

That fixture is the one test_loading_incomplete_pe_file pins, and #663 already decided a truncated file's missing bytes stay missing. Zero-filling the whole declared extent breaks that test and turns the padding into blocks; backing only the sections the file omits entirely does neither, but still invents bytes.

So either the object should stop claiming addresses it cannot back, or those sections should be zeroed. Neither changes the functions or blocks CFGFast recovers here. Happy to open a pull request for whichever you prefer.

Related: #790.

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Measurement record for the report above. Everything here is on public inputs: 104 PE files, 92 of them fixtures in angr/binaries and 12 from the BSD-2-Clause decbench set.

Reproducer, on 46a3733 unmodified

>>> ld = cle.Loader(".../tests/i386/windows/a94bbeed0ef51db3d3964bb0cc2cbed0adab0e47997d88f34daa92faa1a91e8a", auto_load_libs=False)
>>> hex(ld.main_object.min_addr), hex(ld.main_object.max_addr)
('0x10000', '0x13f29')
>>> ld.main_object.contains_addr(ld.main_object.max_addr)
True
>>> ld.find_object_containing(ld.main_object.max_addr)
None
>>> ld.describe_addr(ld.main_object.max_addr)
'not part of a loaded object'
>>> len(ld.memory.load(ld.main_object.min_addr, ld.main_object.max_addr - ld.main_object.min_addr + 1))
5793
>>> ld.memory.load(ld.main_object.sections_map[".rdata"].vaddr, 1)
KeyError: 77056

The file is 5793 bytes. .text is cut short by the end of file and keeps the part that is there; .rdata, .data, INIT and .reloc all begin past it and are skipped with the "Skipping this section" warning, while .reloc still sets max_addr. Of the 10377-byte gap, 5705 bytes are the tail of the truncated .text — the part #663 rules on — and 4672 are the four sections the file does not contain.

The instrument fires. The same census over the same 104 files on 46a3733 reports all 104 short of their own max_addr, 100 of them by a single byte. With #790 applied, 1 of 104 is still short, and it is this fixture. 104 candidates in and 104 records out on every arm.

Arm A — the object stops claiming addresses it cannot back (max_addr ends at the last backed byte). Against #790 as the baseline: 103 of 104 records byte-identical; 0 images shrank, 0 new load errors, 0 shortfalls gained, 0 objects moved min_addr, 0 mapped image bytes changed. One record changes, the fixture: max_addr 0x13f29 to 0x116a0, describe_addr now names the object. CFGFast(normalize=True) on it: 54 functions, 442 nodes, 629 edges before and after, and the address sets are identical. MemoryData goes from 106 records to 45; every record lost sits above the last backed byte, i.e. at an address nothing can read.

Arm B — zero only the sections the file omits entirely, leaving the truncated .text tail unbacked. 103 of 104 byte-identical, same zeroes on every regression counter, max_addr unchanged, describe_addr fixed. CFGFast is identical on the fixture including MemoryData.

Arm C — zero the whole declared extent. test_loading_incomplete_pe_file fails. CFGFast on the fixture goes from 54 functions to 115 and from 442 nodes to 481; all 39 new blocks and 13 of the new functions are inside the zero fill, and the other 48 are existing blocks promoted to function entries. So the padding manufactures code rather than recovering it.

Tests. The cle suite is 240 passed, 9 skipped on the baseline and on arms A and B alike; one pre-existing environment failure, tests/test_clemory.py::test_cclemory, a cffi compile error, is identical on the unmodified tree. Arm C adds test_loading_incomplete_pe_file. In angr, tests/analyses/test_unpacking.py — whose test_packing_detection_a94b loads this exact fixture — is 7 passed under the baseline and under both arms, as are tests/analyses/cfg/test_iat_resolver.py and tests/analyses/cfg/test_pe_meta_structs.py.

Both arms exist as local branches on top of #790 and are a few lines each; say which you want and I will open it.