Skip to content

newt: Make generated CMakeLists.txt build like newt build - #635

Draft
andrzej-kaczmarek wants to merge 9 commits into
apache:masterfrom
andrzej-kaczmarek:cmake-fixes
Draft

newt: Make generated CMakeLists.txt build like newt build#635
andrzej-kaczmarek wants to merge 9 commits into
apache:masterfrom
andrzej-kaczmarek:cmake-fixes

Conversation

@andrzej-kaczmarek

@andrzej-kaczmarek andrzej-kaczmarek commented Oct 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

The CMakeLists.txt generated by newt target cmake failed to link on a clean tree. When it did link, it produced a different image than newt build. This PR reworks the generator so that a CMake build matches newt build, and fixes three issues found along the way that also affect newt build itself.

Fixes affecting newt build

  • Deterministic link tables order: link tables were collected by iterating over a map, so link_tables.ld.h, and with it the image layout, could change between builds of the same target.
  • Arguments in compile_commands.json: commands were joined with spaces without quoting, so arguments containing quotes or spaces (e.g. -DMBEDTLS_USER_CONFIG_FILE="mbedtls/config_mynewt.h") were parsed differently by tools than they were passed to the compiler. Each command is now written as a list of arguments, which needs no quoting on any platform.
  • Data race on compile commands: files of a package are compiled by multiple workers using the same compiler object, but compile commands were collected without the compiler mutex, so entries of compile_commands.json could be lost.

Refactoring (no functional change)

These commits expose what newt computes internally, so the CMake generator can use exactly the same values:

  • compile/link flags, include directories, link options and the objcopy command (toolchain.Compiler),
  • generation of build inputs before compilation (key file, link tables, pre-build scripts),
  • command line, environment and working directory of user scripts, plus selection of the script interpreter on Windows,
  • directories whose archives/object lists are passed to the linker,
  • destination path of copied pre-compiled archives.

CMake generator

  • runs the newt pre-compile steps (link tables, key file, pre-build commands) when generating the file,
  • compiles each package with exactly the flags and include directories newt uses, in the same order, ignoring flags CMake takes from the environment (CFLAGS, LDFLAGS, ...) and the build type, and writes archives to the same paths, so linker scripts, external scripts and newt load/debug/create-image work with either build system,
  • runs pre-build, pre-link and post-link commands at the same stages and with the same environment as newt, with a new work directory for each build, and links .a files they emit,
  • finds pre-compiled archives not named lib*.a, also in source directories of other repositories,
  • keeps __FILE__ relative, as newt does,
  • works out the linker inputs at build time the same way newt does and passes them in a response file, relinking only when one of them or a linker script changes,
  • links with the same command line as newt and generates the .bin, .lst and .map files on every build,
  • fails explicitly for split images and multi-config generators, which are not supported.

Testing

Images built with the generated CMakeLists.txt (Ninja) were compared with newt build for blinky (nRF5340), bleprph and bleprph_oic (nRF52840), btshell (nRF5340), mcuboot (FRDM-K64F) and blinky (NUCLEO-U575ZI-Q). The .bin files, symbols and sections are identical; only debug info differs because CMake passes absolute source paths. For btshell, the .bin files differ only inside the network core image, which its pre-link command rebuilds with a new timestamp on every build. A no-op rebuild does not relink. bleprph was also tested the same way on Windows, with newt running in MSYS2 (UCRT64).

🤖 Generated with Claude Code

@andrzej-kaczmarek
andrzej-kaczmarek marked this pull request as draft October 16, 2026 08:10
@andrzej-kaczmarek
newt/builder: Make link tables order deterministic
dd3ea99 
Link tables were collected by iterating over a map of packages, so
their order in link_tables.ld.h, and thus the layout of the linked
image, could change between builds of the same target. Sort packages
by name before collecting link tables. Also close the generated file.

Assisted-by: Claude:claude-opus-5-5
@andrzej-kaczmarek
newt/toolchain: Write arguments list to compile_commands.json
15364a8 
Compile commands were joined with spaces without any quoting, so
arguments containing quotes or spaces, e.g.
-DMBEDTLS_USER_CONFIG_FILE="mbedtls/config_mynewt.h", were parsed
differently by tools reading compile_commands.json than they were
passed to the compiler. Write each command as a list of arguments
instead of a command line, which doesn't need quoting on any
platform.

Assisted-by: Claude:claude-opus-5-5
@andrzej-kaczmarek
newt/toolchain: Fix data race on compile commands
647265b 
Source files of a package are compiled by multiple build workers using
the same compiler object, but the list of compile commands written to
compile_commands.json was appended to without holding the compiler
mutex, so entries could be lost. Protect it with the mutex, as is done
for the list of object files.

Assisted-by: Claude:claude-opus-5-5
@andrzej-kaczmarek
newt/toolchain: Expose compile and link flags
a7668b2 
Split calculation of compile and link flags out of the functions
building the command lines, so that other build systems generated by
newt can use exactly the same flags:

- CompileFlags() and IncludeDirs() return the flags used to compile a
  source file of a given type,
- LinkFlags() returns the flags preceding and following the input
  libraries on the link command line,
- LinkOptions() returns the artifacts generated when linking,
- BinFileCmd() returns the command converting an elf file to a raw
  binary,
- Builder.linkCompiler() creates the compiler used for linking.

No functional change.

Assisted-by: Claude:claude-opus-5-5
@andrzej-kaczmarek
newt/builder: Extract preparation of build inputs
c8c2337 
Move the steps of TargetBuilder.Build() which generate build inputs
before compilation (key file source, link tables, pre-build user
scripts) to a separate function, so that they can be reused by other
build systems generated by newt.

No functional change.

Assisted-by: Claude:claude-opus-5-5
@andrzej-kaczmarek
newt/builder: Extract preparation of user script commands
ec42414 
Split calculation of the command line, environment and working
directory of a user script from its execution, and move selection of
the shell script interpreter on Windows to a separate function, so
that other build systems generated by newt can execute user scripts
the same way newt does.

A script path relative to the package directory is now made absolute.
Since scripts are executed from the package directory, this doesn't
change which file is executed.

Assisted-by: Claude:claude-opus-5-5
@andrzej-kaczmarek
newt/builder: Extract calculation of linker input directories
98333a2 
Separate the rule selecting the directories, whose archives and object
lists are passed to the linker, from globbing them at link time, so
that other build systems generated by newt can pass the same files to
the linker. Also drop an unused variable.

No functional change.

Assisted-by: Claude:claude-opus-5-5
@andrzej-kaczmarek
newt/toolchain: Expose destination path of copied archives
29358d4 
Add a function returning the path a pre-compiled archive found in
package sources is copied to, so that other build systems generated by
newt can put such archives where newt links them from.

No functional change.

Assisted-by: Claude:claude-opus-5-5
@andrzej-kaczmarek
newt/builder: Generate CMakeLists.txt building like newt build
5eacd0c 
The generated CMakeLists.txt failed to link on a clean tree and,
when it linked, produced a different image than newt build:

- link_tables.ld.h and the key file source were not generated,
- pre-build, pre-link and post-link commands were not executed and
  .a files emitted by them were not linked,
- pre-compiled archives not named lib*.a were not found,
- __FILE__ expanded to absolute paths, making images larger,
- compile flags were sorted together with compiler package flags and
  escaped only partially,
- input files and linker flags were passed in a different order,
- .bin, .lst and .map files were not generated and the elf file was
  written under a different name and location.

Rework the generator so that the build matches newt build:

- run the pre-compile steps of newt build (link tables, key file,
  pre-build commands) when generating the file,
- compile each package with flags and include directories taken from
  the compiler newt uses, ignoring flags CMake takes from the
  environment and build type, and write package archives to the same
  paths newt uses, so linker scripts, external scripts and newt
  load/debug/create-image work with either build system,
- execute external commands at the same build stages and with the
  same environment as newt, using a new work directory for each build,
- calculate the linker input files at build time the same way newt
  does and pass them to the linker in a response file, relinking only
  when any of them or a linker script changes,
- link with the same command line as newt and generate .bin, .lst
  and .map files on every build,
- fail explicitly for split images and multi-config generators,
  which are not supported.

Assisted-by: Claude:claude-opus-5-5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kasjer kasjer Awaiting requested review from kasjer
@m-gorecki m-gorecki Awaiting requested review from m-gorecki

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@andrzej-kaczmarek