Skip to content

fix: send status events faster, in batches, and show what was not sent - #24

Merged
padget11 merged 3 commits into
mainfrom
fix/publish-event-faster-sign-in
Oct 29, 2026
Merged

padget11 merged 3 commits into
mainfrom
fix/publish-event-faster-sign-in

Conversation

@DaRoszko

Copy link
Copy Markdown
Contributor

What changed

scripts/publish-event.sh, same inputs and outputs:

  1. Token exchange with one curl to https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token (client credentials, client_id = AIP_EVENTS_CLIENT_ID, the job's OIDC token as client_assertion, read from a file, scope = {AIP_EVENTS_AUDIENCE}/.default), replacing az login and az account get-access-token. No AZURE_CONFIG_DIR, and the OIDC token is no longer on a command line. A refusal names Entra's code, for example (AADSTS700213).
  2. aws sqs send-message-batch, up to 10 events a call, each keeping its FIFO group and deduplication ID.
  3. Only transient failures are retried: curl --retry (timeouts, 408, 429, 5xx), AWS_RETRY_MODE=standard with AWS_MAX_ATTEMPTS=3, and batch entries SQS failed on its side. A refused token, a refused role or a sender fault is final.
  4. The events file is parsed once, not three jq calls per event.
  5. The run's summary page says sent N of M, and names each event not sent and why, as well as the existing warnings.

Why

In DEFRA/ai-platform-deployments run 40851790376 the first publish spent its whole 10 s budget on az login and get-access-token and sent nothing. In the second, az took about 6.8 s before the first STS call. Each AWS CLI call takes about 0.9 s, so one call per event does not fit an apply with several operations, or reconcile-all, inside the budget.

Ordering and duplicates: each event is its own FIFO group (operation, commit or environment) and its ID includes group, status, run and attempt, so one publish never sends two events of one group, and a resent entry keeps its deduplication ID.

Verification

  • tests/integration/scripts/publish-event.test.js: 35 passed (Git Bash, Windows). New cases: the token exchange form fields and assertion file, batches of 10, retry settings, Entra's refusal reason, non-JSON answers, only SQS-side entry failures resent, sender faults not resent, a failed batch call not resent, a refused role not retried, the summary page.
  • npm run lint and shellcheck scripts/publish-event.sh clean.
  • Not yet run against Entra and SQS: the next infradev apply after this release is the proof (expect Sent N of N events on the summary page).

Impact

Patch (fix:): callers pass the same variables. AIP_PUBLISH_RETRIES now caps the transient retries of curl, the AWS CLI and failed batch entries. docs/async-operations-plan.md in ai-platform-discovery-docs still says azure/login; that is a separate docs PR.

publish-event exchanges the job's OIDC token at Entra's token endpoint with one curl instead of az login and get-access-token (about 6 s of the 10 s budget in infradev), sends with send-message-batch (one AWS CLI call per 10 events), retries only transient failures (curl --retry, AWS_RETRY_MODE=standard, and entries SQS failed on its side; a refusal is final), parses the events once, and writes what was and was not sent to the run's summary page.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Batch publishing can fail before the AWS CLI starts when the single argument exceeds Linux’s size limit, while discarded per-entry error details leave refusals unexplained.

2 open issues
What changed in this PR

This PR speeds up status-event publishing for the reusable workflows while retaining the shared deadline and non-blocking deployment behavior.

Changes:

  • Replaces Azure CLI authentication with a direct Entra token exchange.
  • Sends events in batches of up to 10, with transient-failure retries.
  • Adds delivery counts and unsent-event notices to the run summary, with expanded integration coverage.
File Description
tests/​integration/​scripts/​publish-event.test.js Covers token exchange, batching, retries, and summaries.
tests/​helpers/​scripts.js Updates curl and AWS fakes for token exchange and batch responses.
scripts/​publish-event.sh Implements direct authentication, batch delivery, and summary reporting.
README.md Documents the revised authentication and delivery behavior.
.github/​actions/​publish-event/​action.yml Updates the action description to reflect batching and transient retries.

Review effort: Balanced

Comment thread scripts/publish-event.sh Resolved
Comment thread scripts/publish-event.sh Resolved
Ten large events in one --entries argument could pass Linux's per-argument limit, so the batch goes in a file in the protected working directory. Per-entry failures come in a successful response, so the AWS CLI prints nothing: each warning now carries SQS's code and message, on one line and cut to 200 characters. One jq call per attempt instead of four.
@DaRoszko
DaRoszko requested a review from a team as a code owner October 29, 2026 14:42
@DaRoszko
DaRoszko requested a balanced review from Copilot October 29, 2026 14:42

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Count-only SQS batching can exceed the 1 MiB aggregate message-body limit, causing the failed-call branch to abandon an entire batch of individually valid events.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Partition SQS batches by entry count and aggregate message size

scripts/​publish-event.sh:225

SQS limits the combined message bodies in a batch to 1 MiB as well as limiting it to 10 entries. Ten individually valid 110 KiB events exceed that limit, so this count-only partition causes BatchRequestTooLong and the failed-call branch abandons the whole batch. Split batches by both entry count and UTF-8 message-body size, and add an integration test for the aggregate-size boundary.

2 resolved since last review

Review effort: Balanced

The first publish in a job used up the 10 second budget in two of four
infradev applies (runs 40851790376 and 40893670452). Raise the default to
30 seconds and keep it only in src/events.js: build-event already exports
AIP_EVENT_DEADLINE with every events file, so publish-event.sh now keeps to
that deadline alone instead of carrying its own copy of the default, and
sends nothing, with a notice, when it is missing. Comments and the action
description point to the default rather than repeat it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@padget11
padget11 merged commit d0813d1 into main Oct 29, 2026
2 checks passed
@padget11
padget11 deleted the fix/publish-event-faster-sign-in branch October 29, 2026 15:11
ai-platform-orchestrator Bot added a commit that referenced this pull request Oct 29, 2026
🤖 I have created a release *beep* *boop*
---


##
[2.0.3](v2.0.2...v2.0.3)
(2026-10-29)


### Bug Fixes

* send status events faster, in batches, and show what was not sent
([#24](#24))
([d0813d1](d0813d1))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: ai-platform-orchestrator[bot] <333351233+ai-platform-orchestrator[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants