Skip to content

Add a darwin/arm64 Go language-detector fixture - #224

Merged
ltfish merged 2 commits into
masterfrom
sharpen/go-macho-fixture
Sep 4, 2026
Merged

ltfish merged 2 commits into
masterfrom
sharpen/go-macho-fixture

Conversation

@zardus

@zardus zardus commented Sep 3, 2026

Copy link
Copy Markdown
Member

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Problem

There is no Mach-O Go binary here that cle can load, so a loader test for Go on
Mach-O has nothing to run against. tests_src/language_detector/build_go_cross.sh
cross-compiles langdetect_go.go for linux/386, linux/arm, linux/arm64,
windows/386, windows/arm64 and windows/amd64, and for no darwin target.

The one Mach-O Go binary the repository does have is tests/x86_64/terramate.macho,
and it is darwin/amd64:

  File "cle/backends/macho/macho.py", line 827, in _load_lc_unixthread
    raise CLECompatibilityError()
cle.errors.CLECompatibilityError

(cle 2f7657fd.) The Go linker emits LC_UNIXTHREAD for darwin/amd64.

Root cause

The script was written for the architectures a CFG test needed at the time, and
darwin was not one of them.

Fix

Add a darwin/arm64 target and the binary it produces,
tests/aarch64/langdetect_go.macho. darwin/amd64 is left out until cle can load
one, and the script says so beside the line.

Built with go1.27.1, the script's GO_CURRENT, from the same
tests_src/language_detector/langdetect_go.go every other target uses. Unlike
its siblings this one was not built at /workspace, so it is built with
-trimpath. Without that the binary carries the builder's home directory
through its DWARF; with it the file contains no build path at all and the
recorded command reproduces it byte for byte from any working directory.

Testing

The file is a 2,430,258-byte MH_EXECUTE for CPU_TYPE_ARM64 carrying a
__TEXT,__gopclntab section. cle's Go pclntab parser reads 1,888 function
entries out of it, and every one of those addresses lands inside __text.
Rebuilding with the command in the script reproduces the committed bytes
exactly, from two different working directories.

Needed by the cle change that reads the Go pclntab on Mach-O and PE, which adds
a Mach-O regression test against this file.

Validation: #224 (comment)

session: sharpen

build_go_cross.sh covers linux and windows but no darwin, so there is no Mach-O
Go binary here that cle can load: the only one, tests/x86_64/terramate.macho, is
darwin/amd64, where the Go linker emits LC_UNIXTHREAD and cle raises
CLECompatibilityError.

Built with go1.27.1, the script's current toolchain, from the same
langdetect_go.go every other target uses. Unlike its siblings this one was not
built at /workspace, so it is built with -trimpath and carries no build paths at
all; the command reproduces it byte for byte from any directory.
@zardus

zardus commented Sep 3, 2026

Copy link
Copy Markdown
Member Author

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Validation record for head 6b8841aa4c2d87144d3ca2f2a7564ecdd3d5d2c7 against baseline a2eb7cf15c0e5f68baffc94e873f5b6addf5e2d4.

The file. tests/aarch64/langdetect_go.macho, 2,430,258 bytes, SHA-256
4a202cc2158d1d853a0096ee9c4dc56cdd9dd856133eeae6a7c8066b70acb29e. Its first
four bytes are cf fa ed fe, which cle reads little-endian as 0xfeedfacf,
MH_MAGIC_64; cputype 0x0100000C (CPU_TYPE_ARM64), filetype MH_EXECUTE.
It carries __TEXT,__gopclntab, __go_buildinfo, __go_func, __go_module,
__go_type, __noptrdata and __noptrbss.

Reproducibility. build_go_cross.sh exports CGO_ENABLED=0 GOTOOLCHAIN=local and runs, for this target:

GOOS=darwin GOARCH=arm64 "$GO_SDK_DIR/go1.27.1/bin/go" build -trimpath \
    -o "$ROOT/tests/aarch64/langdetect_go.macho" \
    "$ROOT/tests_src/language_detector/langdetect_go.go"

Both paths are absolute, so the command runs from anywhere. Run from the
repository root and again from /tmp, with GOCACHE and GOPATH pointing
somewhere fresh, it produces
4a202cc2158d1d853a0096ee9c4dc56cdd9dd856133eeae6a7c8066b70acb29e both times,
byte for byte equal to the committed file.

No build paths. strings finds zero occurrences of the builder's home
directory, of any SDK path, or of the string /home/ in the file. Without
-trimpath the same build embeds the absolute source path and the whole Go
standard library's DWARF paths; the sibling fixtures avoid that by having been
built at /workspace, which is not reproducible here.

What reads it. cle's Go pclntab parser recovers 1,888 function entries from
__TEXT,__gopclntab, with ptr_size 8, min_lc 4 and the go1.20+ header
layout. All 1,888 addresses lie inside __text, and the entries are
monotonically increasing with non-zero sizes. Before this file, no Mach-O Go
binary in the repository loaded at all: tests/x86_64/terramate.macho raises
CLECompatibilityError from MachO._load_lc_unixthread.

Script. bash -n tests_src/language_detector/build_go_cross.sh passes. The
new target follows the existing output-path convention, tests/<arch>/, and the
documented output list at the top of the script gains the matching line.

session: sharpen

@zardus

zardus commented Sep 3, 2026

Copy link
Copy Markdown
Member Author

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

What a Mach-O loader test has to work with, before and after this fixture.

Before — the only Mach-O Go binary in the repository is darwin/amd64, where
the Go linker emits LC_UNIXTHREAD, and cle does not load it:

angr/binaries master
tests/x86_64/langdetect_go.macho     does not exist
tests/aarch64/langdetect_go.macho    does not exist

the one Mach-O Go binary that does exist, and what cle makes of it:

    >>> cle.Loader('tests/x86_64/terramate.macho', auto_load_libs=False)
    File "cle/backends/macho/macho.py", line 827, in _load_lc_unixthread
    raise CLECompatibilityError()
    cle.errors.CLECompatibilityError

After — the new file loads, and the cle branch that reads the Go pclntab on
Mach-O and PE finds a full function table in it:

with this fixture
file size      2430258 bytes
magic          bytes cf fa ed fe -> 0xfeedfacf   (MH_MAGIC_64)
cputype        0x100000c   (CPU_TYPE_ARM64)
filetype       2   (MH_EXECUTE)

sections:
    __text               0x0100001000 0x9c444
    __symbol_stub1       0x010009d460 0x2b8
    __rodata             0x010009d720 0x9ec2
    __gopclntab          0x01000a75e8 0x9d411
    __rodata             0x0100148000 0x1c60
    __go_type            0x0100149c60 0x21528
    __go_func            0x010016b188 0x3c0
    __got                0x010016b548 0x1d0
    __go_buildinfo       0x010016c000 0xf0
    __go_fipsinfo        0x010016c100 0x78
    __go_module          0x010016c180 0x238
    __noptrdata          0x010016c3c0 0x4b42
    __data               0x0100170f20 0x4732
    __bss                0x0100175660 0x26b58
    __noptrbss           0x010019c1c0 0x4030
    __zdebug_abbrev      0x01001a4000 0x15e
    __zdebug_line        0x01001a415e 0x239ae
    __zdebug_frame       0x01001c7b0c 0x7200
    __debug_gdb_scri     0x01001ced0c 0x18
    __zdebug_info        0x01001ced24 0x54ef7
    __zdebug_loc         0x0100223c1b 0x25405
    __zdebug_ranges      0x0100249020 0xc8a9

gopclntab      <GoPclntab: 1888 functions, text at 0x100001000>
    ptr_size   8
    min_lc     4
    go_version (1, 20)
    entries whose address is inside an executable section: 1888/1888
    first      go:buildid @ 0x100001000
    last       go:textfipsend @ 0x10009d430

@zardus

zardus commented Sep 3, 2026

Copy link
Copy Markdown
Member Author

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

cle master is red, and this branch holds one of the two fixtures it is waiting
on.

angr/cle#808 merged at 2026-09-03T17:44:30Z as
b6ff025bdc9e15d80c26e86eaf8a61ab1066fd8b. Its two new Mach-O tests load
tests/aarch64/langdetect_go.macho, which exists only on this branch --
GET /repos/angr/binaries/contents/tests/aarch64/langdetect_go.macho at
binaries master 3de2c41a297606c45be7985aa09b3ae615424d61 returns 404. So on
cle master they fail:

FAILED tests/test_gopclntab.py::TestGoPclntab::test_macho_binary - cle.errors.CLEFileNotFoundError: Could not find file .../binaries/tests/aarch64/langdetect_go.macho
FAILED tests/test_gopclntab.py::TestGoPclntab::test_macho_binary_supplies_the_function_symbols - cle.errors.CLEFileNotFoundError: Could not find file .../binaries/tests/aarch64/langdetect_go.macho

Read from cle CI run 33786713158, job Test macos-15 (id 100753147049), at cle
master head 3812052df2ad284cd16684fb7b7eb66e8d14dc6d; the ... stands in for
/Users/runner/work/cle/cle/cle/tests/../... The same two failures appear in
that run's Test (Pyodide) job and in its ci / Test (1) shard, and in the
earlier run 33786341934 at b6ff025bdc9e15d80c26e86eaf8a61ab1066fd8b. On a
push to master the cle workflow always takes angr/binaries at master --
actions/binaries-ref reads the pull-request body, which a push does not have,
and resolve_refs.py skips its branch match when the branch is master -- so
no branch or pull-request reference can supply the file instead.

Merging this branch clears those two. It does not make cle master green by
itself: angr/cle#787 merged at 2026-09-03T17:48:10Z and needs
#193 for tests/aarch64/relocatable_object.macho. Both binaries
pull requests have to land.

Head here is 5f81209953b00e6786503de45f11a17fde903fd4, which GitHub reports
mergeable and clean against binaries master
3de2c41a297606c45be7985aa09b3ae615424d61, and not behind it.

@ltfish
ltfish merged commit 003e82a into master Sep 4, 2026
zardus added a commit that referenced this pull request Sep 6, 2026
cle's tests on master now load tests/aarch64/langdetect_go.macho and
tests/aarch64/relocatable_object.macho, which #193 and #224 added after this
branch was cut. angr/cle#807 names this pull request in its sync: line, so CI
checks this branch out instead of master and those two files were missing:
3 failed, 264 passed on the macOS job. Merging master in supplies them and
leaves this branch's own five objects and build script untouched.
zardus added a commit that referenced this pull request Sep 6, 2026
cle's tests on master now load tests/aarch64/langdetect_go.macho and
tests/aarch64/relocatable_object.macho, which #193 and #224 added after this
branch was cut. angr/cle#764 and angr/cle#804 name this pull request in their
sync: lines, so CI checks this branch out instead of master; once either is
rebased onto current cle master those two files would be missing and the macOS
job would fail. Merging master in supplies them and leaves this branch's own
three objects and build script untouched.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants