wip(ci_visibility): preserve pytest log delivery

[gnufede]

Description

Fix the closed-stream shutdown error in #16712 without the whole-process loss of tracer log delivery exposed by the compatibility probes for #20101.

• Replace the unconditional ddtrace propagation cutoff with per-handler filters installed during pytest teardown. Only ddtrace records destined for already-closed ordinary StreamHandlers are skipped; healthy root and direct handlers continue receiving records.
• Rescan at session finish, unconfigure, and final cleanup, including when instrumentation is disabled. Preserve propagation settings, handler ownership, formatting, application records, and the global tracer's lifetime.
• Exclude FileHandler and custom handler subclasses, preserve existing filters, avoid duplicate filters across repeated pytest sessions, and allow delivery again if a handler's stream is replaced.
• Retain the 29 subprocess compatibility/control cases added earlier, replace the implementation-specific propagation assertion with preservation tests, and add shutdown/root-file delivery, late configuration, collection-error, repeated pytest.main(), and filter unit tests.
• Update the existing customer-facing release note to describe preserved diagnostic delivery.

Testing

Python 3.13.13:

• scripts/run-tests --venv 3dc4202 -- -n 0 -k 'test_pytest_log_propagation or test_logging or test_pytest_log_correlation or test_plugin': 181 passed on pytest 7.4.4.
• scripts/run-tests -s --venv 1b6f43f -- -n 0 -k 'test_pytest_log_propagation or test_logging or test_pytest_log_correlation or test_plugin': 181 passed on pytest 8.4.2 in the final restored state.
• All 18 previously failing delivery probes now pass, alongside the original shutdown regressions and application/correlation/submission-handler controls.
• Negative control: temporarily disabled the closed-stream filter; all three targeted fd-capture shutdown tests failed with ValueError: I/O operation on closed file. Restored the filter before the final passing run.
• Python formatting and Ruff checks passed through scripts/lint for all four edited Python files; git diff --check passed.
• Scoped type checking passed for the logging helper and subprocess test module. Wider typing reports existing diagnostics in the plugin, imported dependencies, and an unchanged unreachable statement in test_logging.py. Full lint remains limited by unavailable auxiliary tools, including cython-lint.

Risks

The protection is deliberately limited to existing standard stream handlers at pytest teardown; custom/file handler behavior is unchanged. Handlers installed after pytest returns remain the caller's responsibility. The filter checks whether a stream is already closed at dispatch, not arbitrary concurrent closure between filtering and writing. The submission probe uses the real handler with a mock writer; it does not validate remote intake delivery.

Additional Notes

No global logging monkeypatch, no global exception suppression, and no early shutdown of the global tracer. No commit, push, or remote PR update performed.

---

PR by Bits - View session in Datadog

Comment @DataDog to request changes

[datadog-datadog-prod-us1-2]

View session in Datadog

Bits Code status: ✅ Done

Comment @DataDog to request changes

[datadog-official]

I can only run on private repositories.

[cit-pr-commenter-54b7da]

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

ddtrace/testing/internal/logging.py                                     @DataDog/ci-app-libraries
ddtrace/testing/internal/pytest/plugin.py                               @DataDog/ci-app-libraries
releasenotes/notes/fix-pytest-log-propagation-closed-file-3dcb4cc3c51c967f.yaml  @DataDog/apm-python
tests/testing/internal/pytest/test_pytest_log_propagation.py            @DataDog/ci-app-libraries
tests/testing/internal/test_logging.py                                  @DataDog/ci-app-libraries

[cit-pr-commenter-54b7da]

Circular import analysis

⚠️ Existing circular imports

There are 3 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.llmobs -> ddtrace.llmobs._evaluators -> ddtrace.llmobs._evaluators.format -> ddtrace.llmobs._experiment -> ddtrace.llmobs
ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector
ddtrace.appsec._asm_request_context -> ddtrace.appsec._iast._iast_request_context_base -> ddtrace.appsec._iast._iast_env -> ddtrace.appsec._iast.reporter -> ddtrace.appsec._exploit_prevention.stack_traces -> ddtrace.appsec._asm_request_context

[cit-pr-commenter-54b7da]

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 230 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 230 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=134)
ddtrace.internal.opentelemetry.trace -×-> ddtrace.trace  (product:opentelemetry -> product:tracing, score=132)
ddtrace.internal.opentelemetry.context -×-> ddtrace.trace  (product:opentelemetry -> product:tracing, score=132)
ddtrace.appsec._contrib.django -×-> ddtrace.trace  (product:appsec -> product:tracing, score=132)
ddtrace.llmobs._integrations.bedrock_agents -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

[datadog-datadog-prod-us1-2]

More details

The filter blocks ddtrace records only when a standard stream handler has a closed stream. It keeps log delivery to all open handlers.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 6dc5f3a · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}