test(nodejs): reduce redundant weblog scenario coverage - #7689
Conversation
|
|
73ce161 to
0087151
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0087151ad0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| supported_scenarios: | ||
| - DEFAULT | ||
| - GRAPHQL_APPSEC | ||
| - GRAPHQL_ERROR_TRACKING |
There was a problem hiding this comment.
When the Node.js AI_GUARD* scenarios run, AIGuardScenario selects dd_trace weblogs, but this change removes that category from express4 without explicitly listing any of the five AI Guard scenarios. The orchestrator therefore schedules them only on express5, while manifests/nodejs.yml:110-213 marks every non-express4 weblog irrelevant for these classes, so most Node.js AI Guard tests are skipped and their feature coverage disappears. Add the AI Guard scenarios here or retarget those manifest declarations to a scheduled weblog. .cursor/rules/pr-review.mdcL67-L69
Useful? React with 👍 / 👎.
| if library == "nodejs": | ||
| continue |
There was a problem hiding this comment.
When a future Node.js Dockerfile is added without explicit metadata, WeblogMetaData.load() synthesizes an entry with no categories or supported scenarios, so test_all_weblog_has_metadata still sees its name and passes even though the orchestrator silently removes it from every job. The repository documentation identifies this legacy parity test as the guard against exactly that failure, but this blanket branch disables it for all current and future Node.js weblogs; narrow the exception to the intentional matrix differences or add an explicit nonempty-selection assertion instead. .cursor/rules/pr-review.mdcL14-L18
Useful? React with 👍 / 👎.
| supported_scenarios: | ||
| - DEFAULT | ||
| - GRAPHQL_APPSEC | ||
| - GRAPHQL_ERROR_TRACKING |
There was a problem hiding this comment.
When FEATURE_FLAGGING_AND_EXPERIMENTATION is selected for Node.js, its dd_trace category now schedules only express5, but the file-level declarations at manifests/nodejs.yml:1830-1854 mark every weblog except express4 as incomplete_test_app. As a result, the dynamic-evaluation, exposure, and metric suites are all skipped in the workflow step at .github/workflows/run-end-to-end.yml:315-317. Add this scenario to express4 or transfer the manifest support to the weblog that remains scheduled. .cursor/rules/pr-review.mdcL67-L69
Useful? React with 👍 / 👎.
| supported_scenarios: | ||
| - DEFAULT | ||
| - GRAPHQL_APPSEC | ||
| - GRAPHQL_ERROR_TRACKING |
There was a problem hiding this comment.
The OTEL_SEMANTICS_OTLP and OTEL_SEMANTICS_OTLP_TRACE_METRICS scenarios also select dd_trace weblogs, so this list leaves only express5 in their Node.js jobs. However, manifests/nodejs.yml:2897-2916 marks all weblogs missing for the supported server, client, and trace-metric classes except express4; those validations are therefore skipped despite the dedicated workflow steps at .github/workflows/run-end-to-end.yml:539-550. Explicitly retain express4 for the supported OTLP semantics scenarios or update the manifest to a scheduled compatible weblog. .cursor/rules/pr-review.mdcL67-L69
Useful? React with 👍 / 👎.
| supported_scenarios: &framework_scenarios | ||
| - DEFAULT |
There was a problem hiding this comment.
When the Node.js TRACING_CONFIG_EMPTY job runs, fastify no longer matches because its dd_trace category was replaced by this AppSec-only list, leaving only express5 scheduled. The manifest entry at manifests/nodejs.yml:2743-2746 marks every weblog except fastify as having an incomplete /otel_drop_in_baggage_api_datadog endpoint, so Test_Baggage_Headers_Api_Datadog is now skipped entirely. Add TRACING_CONFIG_EMPTY to fastify or move that endpoint's manifest support to a scheduled weblog. .cursor/rules/pr-review.mdcL67-L69
Useful? React with 👍 / 👎.
| express4: | ||
| build_mode: local | ||
| categories: [dd_trace, dd_trace_graphql] | ||
| supported_scenarios: |
There was a problem hiding this comment.
Can you get the approval from all impacted team?
Another point, by using an inclusion list, it'll be easy to not activate a scenario by error, which is a QA issue. By using using excluded_scenarios, the tradeoff will be in favor of QA, as it'll be easy to include a scenario by error, which slows the CI, but at the benefit of never forgot a scenario by mistake.
|
|
||
|
|
||
| @scenarios.test_the_test | ||
| def test_legacy_scenario_matrix(): |
There was a problem hiding this comment.
This test is a leftoever, I'll remove it in another PR.
3 participants
Motivation
The Node.js end-to-end matrix runs most scenarios against
express4,express4-typescript,express5,fastify,nextjs, anduds-express4.This repeats the same coverage across weblogs and creates excessive jobs when
dd-trace-js uses a 120-second shard target.
Across the six core weblogs, the change reduces scenario-weblog executions
from 595 to 170. At the 120-second target, uncapped demand drops from 302 to
83 jobs overall; the existing matrix hits the 256-job orchestrator ceiling.
Changes
express5as the primary weblog for fulldd_tracescenario coverage.express4for default and GraphQL-specific coverage.express4-typescriptto debugger and IAST coverage relevant to thatweblog.
fastifyandnextjsto representative AppSec and IAST scenarios.uds-express4to UDS-relevant agent and telemetry scenarios.Validation
venv/bin/ruff check tests/test_the_test/test_ci_orchestrator.pyvenv/bin/yamllint utils/build/docker/nodejs/weblog_metadata.ymlvenv/bin/python utils/manifest/validate.pygit diff --checkWorkflow
R&P team
🚀 Once your PR is reviewed and the CI green, you can merge it!
🛟 #apm-shared-testing 🛟
Reviewer checklist
tests/ormanifests/is modified? I have the approvalfrom R&P team
build-XXX-imagelabel is presentR&P team
Generated by Codex.