You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix: send status events faster, in batches, and show what was not sent - #24
scripts/publish-event.sh, same inputs and outputs:
Token exchange with one curl to https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token (client credentials, client_id = AIP_EVENTS_CLIENT_ID, the job's OIDC token as client_assertion, read from a file, scope = {AIP_EVENTS_AUDIENCE}/.default), replacing az login and az account get-access-token. No AZURE_CONFIG_DIR, and the OIDC token is no longer on a command line. A refusal names Entra's code, for example (AADSTS700213).
aws sqs send-message-batch, up to 10 events a call, each keeping its FIFO group and deduplication ID.
Only transient failures are retried: curl --retry (timeouts, 408, 429, 5xx), AWS_RETRY_MODE=standard with AWS_MAX_ATTEMPTS=3, and batch entries SQS failed on its side. A refused token, a refused role or a sender fault is final.
The events file is parsed once, not three jq calls per event.
The run's summary page says sent N of M, and names each event not sent and why, as well as the existing warnings.
Why
In DEFRA/ai-platform-deployments run 40851790376 the first publish spent its whole 10 s budget on az login and get-access-token and sent nothing. In the second, az took about 6.8 s before the first STS call. Each AWS CLI call takes about 0.9 s, so one call per event does not fit an apply with several operations, or reconcile-all, inside the budget.
Ordering and duplicates: each event is its own FIFO group (operation, commit or environment) and its ID includes group, status, run and attempt, so one publish never sends two events of one group, and a resent entry keeps its deduplication ID.
Verification
tests/integration/scripts/publish-event.test.js: 35 passed (Git Bash, Windows). New cases: the token exchange form fields and assertion file, batches of 10, retry settings, Entra's refusal reason, non-JSON answers, only SQS-side entry failures resent, sender faults not resent, a failed batch call not resent, a refused role not retried, the summary page.
npm run lint and shellcheck scripts/publish-event.sh clean.
Not yet run against Entra and SQS: the next infradev apply after this release is the proof (expect Sent N of N events on the summary page).
Impact
Patch (fix:): callers pass the same variables. AIP_PUBLISH_RETRIES now caps the transient retries of curl, the AWS CLI and failed batch entries. docs/async-operations-plan.md in ai-platform-discovery-docs still says azure/login; that is a separate docs PR.
publish-event exchanges the job's OIDC token at Entra's token endpoint with one curl instead of az login and get-access-token (about 6 s of the 10 s budget in infradev), sends with send-message-batch (one AWS CLI call per 10 events), retries only transient failures (curl --retry, AWS_RETRY_MODE=standard, and entries SQS failed on its side; a refusal is final), parses the events once, and writes what was and was not sent to the run's summary page.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Batch publishing can fail before the AWS CLI starts when the single argument exceeds Linux’s size limit, while discarded per-entry error details leave refusals unexplained.
Ten large events in one --entries argument could pass Linux's per-argument limit, so the batch goes in a file in the protected working directory. Per-entry failures come in a successful response, so the AWS CLI prints nothing: each warning now carries SQS's code and message, on one line and cut to 200 characters. One jq call per attempt instead of four.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Count-only SQS batching can exceed the 1 MiB aggregate message-body limit, causing the failed-call branch to abandon an entire batch of individually valid events.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Partition SQS batches by entry count and aggregate message size
scripts/publish-event.sh:225
SQS limits the combined message bodies in a batch to 1 MiB as well as limiting it to 10 entries. Ten individually valid 110 KiB events exceed that limit, so this count-only partition causes BatchRequestTooLong and the failed-call branch abandons the whole batch. Split batches by both entry count and UTF-8 message-body size, and add an integration test for the aggregate-size boundary.
The first publish in a job used up the 10 second budget in two of four
infradev applies (runs 40851790376 and 40893670452). Raise the default to
30 seconds and keep it only in src/events.js: build-event already exports
AIP_EVENT_DEADLINE with every events file, so publish-event.sh now keeps to
that deadline alone instead of carrying its own copy of the default, and
sends nothing, with a notice, when it is missing. Comments and the action
description point to the default rather than repeat it.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
scripts/publish-event.sh, same inputs and outputs:curltohttps://login.microsoftonline.com/{tenant}/oauth2/v2.0/token(client credentials,client_id=AIP_EVENTS_CLIENT_ID, the job's OIDC token asclient_assertion, read from a file,scope={AIP_EVENTS_AUDIENCE}/.default), replacingaz loginandaz account get-access-token. NoAZURE_CONFIG_DIR, and the OIDC token is no longer on a command line. A refusal names Entra's code, for example(AADSTS700213).aws sqs send-message-batch, up to 10 events a call, each keeping its FIFO group and deduplication ID.curl --retry(timeouts, 408, 429, 5xx),AWS_RETRY_MODE=standardwithAWS_MAX_ATTEMPTS=3, and batch entries SQS failed on its side. A refused token, a refused role or a sender fault is final.jqcalls per event.sent N of M, and names each event not sent and why, as well as the existing warnings.Why
In DEFRA/ai-platform-deployments run 40851790376 the first publish spent its whole 10 s budget on
az loginandget-access-tokenand sent nothing. In the second,aztook about 6.8 s before the first STS call. Each AWS CLI call takes about 0.9 s, so one call per event does not fit an apply with several operations, orreconcile-all, inside the budget.Ordering and duplicates: each event is its own FIFO group (operation, commit or environment) and its ID includes group, status, run and attempt, so one publish never sends two events of one group, and a resent entry keeps its deduplication ID.
Verification
tests/integration/scripts/publish-event.test.js: 35 passed (Git Bash, Windows). New cases: the token exchange form fields and assertion file, batches of 10, retry settings, Entra's refusal reason, non-JSON answers, only SQS-side entry failures resent, sender faults not resent, a failed batch call not resent, a refused role not retried, the summary page.npm run lintandshellcheck scripts/publish-event.shclean.Sent N of N eventson the summary page).Impact
Patch (
fix:): callers pass the same variables.AIP_PUBLISH_RETRIESnow caps the transient retries of curl, the AWS CLI and failed batch entries.docs/async-operations-plan.mdin ai-platform-discovery-docs still saysazure/login; that is a separate docs PR.