ci: Ship trtexec in the QA image when built on an inference-runtime base

[devin-ai-integration]

What does the PR do?

Restores a working /usr/src/tensorrt/bin/trtexec in the QA/devel image. L0_device_memory_tracker, L0_memory_growth, L0_perf_resnet and L0_perf_deeprecommender use it.

• In the cibase stage, the real binary is staged with cp -L /usr/src/tensorrt/bin/trtexec, along with the libnvinfer_builder_resource*.so* libraries that trtexec dlopens to build engines.
• In the final stage, the binary is installed at /usr/src/tensorrt/bin/trtexec if the runtime base lacks it. The builder resource libraries are copied in no-clobber (cp -an), then ldconfig runs. The build fails if ldd reports any missing library.
• Bases without trtexec (e.g. CPU-only) are skipped rather than failing.

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

• build
• ci
• docs
• feat
• fix
• perf
• refactor
• revert
• style
• test

Related PRs:

Supersedes #9005.

Where should the reviewer start?

Dockerfile.QA: the new RUN at the end of the cibase stage, and the COPY/RUN after the go installs in the final stage.

Test plan:

• pre-commit passes locally.

• Pipeline 73321996 (first commit only): py3-devel built, and trtexec ran (TensorRT v110400). L0_device_memory_tracker--base then failed with Unable to load library: libnvinfer_builder_resource_sm80.so.11.4.0. inference-runtime omits the builder resources and ldd can't see dlopen'd libraries, so the second commit stages those libraries too.

• Pipeline 73338571 (both commits): scoped py3-base/sdk/devel (x86 + sbsa) + L0_device_memory_tracker--base. All 11 jobs passed.

• CI Pipeline ID: 73338571

Caveats:

• The builder resource libraries are large (one per SM arch), so the devel/QA image grows. This only affects the CI test image, not the shipped runtime image.
• The runtime base and cibase must carry the same TensorRT version. Today both come from the same CUDA release tag.
• The other trtexec users (L0_memory_growth, L0_perf_*) weren't run in the scoped pipeline.

Background

• Pre-break: the final QA stage's $BASE_IMAGE runtime layer was the CUDA devel image (--image=base). That image installs TensorRT with DEVEL=1: libnvinfer-bin (/usr/bin/trtexec, plus the /usr/src/tensorrt/bin/trtexec symlink) and the full builder libraries.
• What broke it: since fix: Build on cuda-dl-base devel and ship on inference-runtime #8994 (GitLab !1896), the runtime layer is the CUDA inference-runtime image. That image has only the TensorRT runtime libraries: no libnvinfer-bin and no builder resources.
• Why ci: copy TensorRT tools into the QA image from cibase #9005 failed: in GitLab pipeline 72557890, COPY --from=cibase /usr/src/tensorrt brought over only the symlink, which dangles without /usr/bin/trtexec. test -x then failed.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• Resolves TRI-2014

Link to Devin session: https://nvidia-cloud.devinenterprise.com/sessions/80a180b0212949f9be74e5230ea9e554
Open in Devin Desktop: https://nvidia-cloud.devinenterprise.com/desktop/session/80a180b0212949f9be74e5230ea9e554?variant=devin

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Confidence Score: 4/5

[High impact] Modifies the QA container build configuration.

The PR should not merge while CPU-only QA image builds still fail.

Findings

1.  CPU-only QA builds fail ▶

Summary

The PR copies trtexec and TensorRT builder libraries from the build image into the QA image.

• QA images now include the tools TensorRT tests need to build engines.

_{Reviews (4) · Last reviewed commit: "ci: Rely on the base image's trtexec sym..." · Reviewed by Greptile}

[mc-nv]

Suggested change

	# The CUDA inference-runtime base ships the TensorRT runtime libraries but not
	# trtexec or the builder resource libraries.
	COPY --from=cibase /opt/trtexec/ /opt/trtexec/
	RUN if [ -e /opt/trtexec/trtexec ]; then \
	if [ ! -e /usr/src/tensorrt/bin/trtexec ]; then \
	mkdir -p /usr/src/tensorrt/bin && \
	mv /opt/trtexec/trtexec /usr/src/tensorrt/bin/trtexec; \
	fi && \
	rm -f /opt/trtexec/trtexec && \
	cp -an /opt/trtexec/. / && \
	ldconfig && \
	! ldd /usr/src/tensorrt/bin/trtexec | grep 'not found'; \
	fi && \
	rm -rf /opt/trtexec
	# The CUDA inference-runtime base ships the TensorRT runtime libraries but not
	# trtexec or the builder resource libraries.
	COPY --from=cibase /usr/bin/trtexec /usr/bin/trtexec

[devin-ai-integration]

0f4e58b switches to a direct COPY --from=cibase /usr/bin/trtexec /usr/bin/trtexec. I couldn't take the suggestion as-is, because on its own it brings back two failures:

• The QA tests (L0_device_memory_tracker, L0_memory_growth, L0_perf_resnet, L0_perf_deeprecommender) call /usr/src/tensorrt/bin/trtexec, which isn't present in inference-runtime. That was the original No such file or directory. The RUN now recreates the devel image's symlink to /usr/bin/trtexec.
• trtexec dlopens libnvinfer_builder_resource_sm*.so.<ver>, which inference-runtime also omits. With only the binary, pipeline 73321996 failed with Unable to load library: libnvinfer_builder_resource_sm80.so.11.4.0. ldd can't detect this because the library is loaded at runtime with dlopen.

So the final stage is now:

COPY --from=cibase /usr/bin/trtexec /usr/bin/trtexec
COPY --from=cibase /usr/lib/*-linux-gnu/libnvinfer_builder_resource*.so.*.*.* /usr/local/tensorrt/builder_resource/
RUN mkdir -p /usr/src/tensorrt/bin && \
    ln -sf /usr/bin/trtexec /usr/src/tensorrt/bin/trtexec && \
    echo /usr/local/tensorrt/builder_resource > /etc/ld.so.conf.d/tensorrt-builder-resource.conf && \
    ldconfig && \
    ! ldd /usr/bin/trtexec | grep 'not found'

The builder resources go into a fixed directory registered with ldconfig, because COPY can't compute the arch-specific /usr/lib/<triplet> destination. The .so.*.*.* glob copies only the real versioned files, not the symlinks to them, so their contents aren't duplicated. A scoped DMT re-validation is running; I'll post the pipeline here when it finishes.

[devin-ai-integration]

Update: 0f4e58b failed in pipeline 73373859. The image built, and trtexec ran from /usr/src/tensorrt/bin, but it still couldn't find the builder resources: Unable to load library: libnvinfer_builder_resource_sm80.so.11.4.0 (job 483674704). Putting them in a separate directory and registering that with ldconfig doesn't work. TensorRT appears to load them only from libnvinfer's own directory, which is where the earlier passing revision (pipeline 73338571) put them.

960aa53 keeps the direct COPY --from=cibase /usr/bin/trtexec, but stages the builder resources in /tmp and moves them into /usr/lib/{x86_64,aarch64}-linux-gnu based on TARGETPLATFORM. Re-validating in pipeline 73388065 (L0 DMT only).

[greptile-apps]

CPU-only QA builds fail

When GPU support is disabled, build.py bases tritonserver_cibase on ubuntu:24.04, which has neither /usr/bin/trtexec nor the builder resource libraries. These unconditional COPY steps now fail instead of skipping the optional files as the previous version did. Restore conditional staging so builds without TensorRT can still produce the QA image.

[mc-nv]

I really think that we need to go with small changes.
Devin overcomplicating the solution IMO.

Suggested change

	# The CUDA inference-runtime base ships the TensorRT runtime libraries but not
	# trtexec or the builder resource libraries it dlopens to build engines. The
	# builder resources must sit next to libnvinfer. QA tests call trtexec through
	# the devel image's /usr/src/tensorrt/bin symlink.
	COPY --from=cibase /usr/bin/trtexec /usr/bin/trtexec
	COPY --from=cibase /usr/lib/*-linux-gnu/libnvinfer_builder_resource*.so.*.*.* /tmp/trt_builder_resource/
	RUN if [ "$TARGETPLATFORM" = "linux/arm64" ]; then \
	TRT_LIB_DIR=/usr/lib/aarch64-linux-gnu; \
	else \
	TRT_LIB_DIR=/usr/lib/x86_64-linux-gnu; \
	fi && \
	mv /tmp/trt_builder_resource/* ${TRT_LIB_DIR}/ && \
	rm -rf /tmp/trt_builder_resource && \
	ls ${TRT_LIB_DIR}/libnvinfer.so.* > /dev/null && \
	mkdir -p /usr/src/tensorrt/bin && \
	ln -sf /usr/bin/trtexec /usr/src/tensorrt/bin/trtexec && \
	ldconfig && \
	! ldd /usr/bin/trtexec | grep 'not found'
	# The CUDA inference-runtime base ships the TensorRT runtime libraries but not
	# trtexec or the builder resource libraries it dlopens to build engines.
	COPY --from=cibase /usr/bin/trtexec /usr/bin/trtexec

[devin-ai-integration]

Agreed it should be smaller. d267df1 trims it to two COPYs and a three-command RUN. The TARGETPLATFORM branch, the /tmp checks and the ldd check are gone. The mv uses /usr/lib/*-linux-gnu/, so it works on both x86 and sbsa.

The single COPY line on its own isn't enough, though. CI has already shown that:

• Binary only (rev 1, pipeline 73321996, job 483227906): trtexec runs but can't create a builder: Unable to load library: libnvinfer_builder_resource_sm80.so.11.4.0. Those libraries aren't in inference-runtime.
• Builder resources outside libnvinfer's directory (rev 3, pipeline 73373859, job 483674704): same error, even with an ld.so.conf.d entry. They have to be next to libnvinfer, which is why the mv is needed (a COPY destination can't take the arch-specific directory).
• The tests call /usr/src/tensorrt/bin/trtexec (L0_device_memory_tracker, L0_memory_growth, L0_perf_*). Without the symlink they fail with the original No such file or directory. The alternative would be changing those test.sh files to call /usr/bin/trtexec directly. Let me know if you'd prefer that.

d267df1 is being validated in pipeline 73388065 (L0 DMT only).

[mc-nv]

there is a broken link that should recover after the above change

$ docker run --rm -it gitlab-master.nvidia.com:5005/dl/dgx/cuda:13.4-inference-runtime-ubuntu24.04--26.10 ls 
lrwxrwxrwx 1 root root 16 Oct  9 07:53 /usr/src/tensorrt/bin/trtexec -> /usr/bin/trtexec

[devin-ai-integration]

Thanks, that makes things simpler. Since the base image already has the symlink, copying /usr/bin/trtexec back fixes the path, and my mkdir/ln -s step was redundant. It would also have broken the build, because ln -s without -f can't replace an existing link. 0a03881 drops it. What remains is the trtexec COPY plus the builder-resource COPY and a one-line mv that puts the builder resources next to libnvinfer:

COPY --from=cibase /usr/bin/trtexec /usr/bin/trtexec
COPY --from=cibase /usr/lib/*-linux-gnu/libnvinfer_builder_resource*.so.*.*.* /tmp/trt_builder_resource/
RUN mv /tmp/trt_builder_resource/* /usr/lib/*-linux-gnu/

The devel builds in pipeline 73388065 hadn't started when I pushed, so that pipeline validates 0a03881 (L0 DMT only).

[devin-ai-integration]

0a03881 passed: all 11 jobs in pipeline 73388065 succeeded, including L0_device_memory_tracker--base (job 483799087). The py3-devel log shows the two COPYs and the single mv, and trtexec built all three engines (vgg19, resnet50, resnet152). That covers the QA-image approach on A100; the DGX-Spark variants haven't run yet. Whether to keep this or move to model generation is your call (see the top-level thread).

[mc-nv]

Honestly I don't really like this approach we coming out with.
Is it the right change? If we need specific set of models to be provided why don't we add it as part of the model generation and use it after in our scenarios?
Another option if we can't use the model generation by some reason, why don't delegate model creation to the job itself instead of feeding fat and heavy Dockerfile.QA image and extend it maintenance?

[devin-ai-integration]

@mc-nv Fair point. Model generation looks like a better place for this than the QA image:

• qa/common/gen_qa_model_repository already runs its TensorRT script inside $TENSORRT_IMAGE (nvcr.io/nvidia/tensorrt:<ver>-py3), and that image ships trtexec.
• The inputs, vgg19_onnx and resnet50_onnx, already come from qa_dynamic_batch_image_model_repository. Model gen could also build vgg19_plan and resnet50_plan there, with the same shapes (min 1, opt/max 32 x3x224x224). L0_device_memory_tracker, L0_memory_growth and L0_perf_* would then copy them like the ONNX models, and Dockerfile.QA wouldn't change.
• Things to confirm: the plans only exist once the model repository is regenerated, and they're tied to the GPU and TRT version, the same as the other plan models model gen already produces.

Building the model inside the job instead would still need trtexec or the TRT builder at test time, so it brings back the same dependency in the job.

I'm holding the Dockerfile change while Rini weighs in. If you agree, I'll rework this PR into the model-gen version.