Repository navigation
fix: resolve recurring NUT failures (run-eval login-rate, publish, cross-platform context-vars) @W-24530265@ - #511
Merged
nico-pappagianis merged 3 commits intoOct 30, 2026
Conversation
- Bump @salesforce/agents to ^2.3.3 to pick up the run-eval JWT-connection fix (SDK PR #382): callEvalApi now routes its SFAP call through the JWT connection like every other SFAP path, instead of the raw refresh-token connection that tripped the per-IP 'login rate exceeded' cap during the long eval POST. - Fan the NUTs across per-group CI jobs (run-eval / test / generate / z / adl-mcp) so each group runs on its own runner IP, keeping any one IP under the login-rate cap. run-eval is isolated for maximum headroom. - De-dup the expensive (~18 min) run-eval invocations: run each spec once in a before() hook and share the parsed result, so a rate-limit failure surfaces once rather than per assertion.
Verify the published agent against the AiAgentDefinition/Bot metadata the publish path now retrieves, and fix the retry-pollution and inter-test _v2 dependency that made the suite fail on mocha retries. Adds --force-overwrite to the generate step so a retried run regenerates cleanly instead of throwing JsonAabExistsError.
- Quote the --context-variables-json payload per-platform (bash strips single quotes, cmd.exe does not), so CV1/CV2/CV3/CV6 pass on Windows as well as Unix. - Remove CV5: it asserted a server-side rejection of an Internal context var, but visibility is purely server behavior (neither plugin flags.ts nor the SDK enforce it) and the server no longer rejects it, so the test pinned a contract the plugin doesn't own.
nico-pappagianis
changed the title
setu4993
approved these changes
Oct 29, 2026
nico-pappagianis
deleted the
fix/nut-suite-run-eval-and-cross-platform
branch
October 30, 2026 17:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Work Item: @W-24530265@
What
Closes out the recurring
plugin-agentNUT failures, all CI-proven green on both ubuntu and windows (validated on the diagnostic branch, run 40917524645). Four independent buckets, one real product fix and three test-side fixes:agent test run-eval(dominant)callEvalApiused the raw refresh-token connection instead of the JWT connection, tripping the per-IPlogin rate exceededcap during the long eval POST@salesforce/agentsto^2.3.3(SDK PR #382)agent publish(z2)_v2dependency; verification not aligned to theAiAgentDefinition/Botmetadata publish now retrieves--force-overwriteon generateagent previewcontext-vars (z3, Windows)--context-variables-jsonis stripped by bash but not cmd.exe, soJSON.parsefailed on Windowsagent previewCV5 (z3)Why
Only the run-eval failure was a real defect, and it lives in the SDK (fixed in
@salesforce/agents@2.3.3); on the plugin side it's a dependency bump. The other three were test-side issues — none wereplugin-agentproduct regressions.Changes
fix(nuts): resolve run-eval login-rate failures— bump@salesforce/agentsto^2.3.3; fan the NUTs across per-group CI jobs (run-eval / test / generate / z / adl-mcp) so each group runs on its own runner IP; de-dup the expensive (~18 min) run-eval invocations into sharedbefore()hooks.test(nuts): fix z2 publish verification for AiAgentDefinition outputtest(nuts): make context-variables-json cross-platform; drop stale CV5Notes for reviewers
.github/workflows/test.yml+ thetest:nuts:*scripts) remains useful beyond run-eval: it keeps per-IP login pressure down for every NUT group. Drop it if you prefer a smaller CI diff.Testing
All four buckets verified green on ubuntu + windows in CI (run 40917524645). The server-behavior change behind CV5 (Internal context vars no longer rejected on preview) was confirmed e2e against a live org and is owned by the server team — intentional, no action needed here.