You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix: defer rate-limited board writes instead of failing the run - #65
The FOC Board Mechanical Rules workflow failed twice on 2026-10-03 (run 37081217551, run 37102189769) during the Cycle rollover: R-FC-013 had hundreds of Cycle updates queued, and after roughly 200 successful writes every later write got 403 Forbidden. That pattern points to GitHub's secondary rate limit, but the logs never captured the response body, so it isn't confirmed. This PR also makes the next failure explain itself.
An earlier version of this PR retried with sleeps inside graphql_query. That was replaced because the retries stacked with set_field_value_bulk's per-item fallback (one throttled batch could block for hours), blocked API server requests for minutes, and ignored 429s.
github-projects-client
graphql_query raises a typed GitHubRateLimitError (403 with a rate-limit message or exhausted quota, or 429) and includes GitHub's own error message in every HTTP error. It never sleeps or retries.
set_field_value_bulk waits 1s between mutation batches (GitHub's guidance for writes). On a rate limit it skips the per-item fallback, stops sending requests, and marks the remaining items rate_limited.
The API server returns 429 for GitHubRateLimitError; without a dedicated handler its GitHubAPIError handler would have returned 502.
foc-mechanical-rules
Rate-limited items are reported as deferred, which doesn't fail the run: the next hourly run retries them. error still fails the run, so problems that need a person (like a missing repo permission) stay loud.
A BatchedFieldRule base class now owns the batched write (group by value, bulk write, defer on rate limit, map results) that R-FC-012/013/014 and R-PR-010 each duplicated; rules only select and decide.
Follow-ups from fix: hint at permissions when R-PR-001 assignee mutation 404s #64: add_assignee returns the resulting assignees, and R-PR-001 now reports an error if GitHub accepted the request without assigning the author (previously this would have shown as "applied"). Its docstring no longer overclaims GitHub's 404 behavior, and the README documents deferred.
GraphQL throttling that GitHub returns as HTTP 200 (RATE_LIMITED errors) is now detected (thanks @rjan90). Detection lives in one public is_rate_limited() covering every shape GitHub documents, including a 403 with Retry-After and the older "abuse detection" wording.
GitHub warns that continuing to send requests while rate limited may get the integration banned. The job's session now has a RateLimitBreaker that sends nothing after the first rate-limited response: remaining items are deferred, later rules are reported not run, and the run doesn't fail.
The CLI saves the mutation log even if a run crashes, so writes that already landed are never forgotten (R-FC-012 relies on that log).
Test plan
github-projects-client: uv run pytest -m "not integration" (102 passed), including rate-limit classification, batch pacing, stop-on-rate-limit with no fallback, non-throttle fallback still working, clear-mode batching, and the server's 429 mapping
foc-mechanical-rules: uv run pytest -m "not integration" (74 passed, including the breaker stopping requests, deferral after a trip, later rules not run, and the mutation log surviving a crash), including deferred results not failing the CLI, error still failing it, and the silent-non-assignment check
ruff lint and format via .githooks/pre-commit
Next Cycle rollover: confirm the run passes with any leftovers listed as deferred, and check the logged GitHub message to confirm the rate-limit diagnosis
R-FC-013's cycle rollover (202610-1 -> 202610-2) queued 211-416 Cycle
field updates in one run, batched into GraphQL mutations of 25 items
each. The first ~200 succeeded within seconds, then GitHub's secondary
rate limit kicked in and returned a flat 403 for the rest of the
batch, failing the whole hourly workflow run
(https://github.com/FilOzone/tpm-utils/actions/runs/37081217551,
.../runs/37102189769).
graphql_query had no retry/backoff at all; any non-2xx just raised
immediately. It now detects a rate-limited 403 (primary or secondary --
both come back as 403 with "rate limit" in the response body's
message, same signal the REST error handler in server/app.py already
uses) and retries up to 3 times, honoring GitHub's Retry-After header
with an exponential-backoff fallback.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The exponential backoff fallback exhausted all 3 retries in 7 seconds
(1s/2s/4s) when neither Retry-After nor rate-limit headers were
present. GitHub's GraphQL rate-limit docs require waiting at least a
minute for the secondary rate limit, and waiting until
X-RateLimit-Reset when the primary rate limit's quota is exhausted
(X-RateLimit-Remaining: 0). Floor and step the backoff at 60s, and
honor X-RateLimit-Reset when applicable, per
#65 (comment).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…sleeping
Retrying inside graphql_query compounded with set_field_value_bulk's
per-item fallback (a throttled 25-item batch could block for hours),
blocked API server requests for minutes, and ignored 429s. Instead:
- graphql_query raises a typed GitHubRateLimitError (403 with a rate
limit message or exhausted quota, or 429) and includes GitHub's own
message in every HTTP error, without retrying.
- set_field_value_bulk waits 1s between mutation batches, and on a rate
limit skips the per-item fallback, stops sending requests, and marks
the remaining items rate_limited.
- foc-mechanical-rules reports those items as "deferred", which does
not fail the run; the next hourly run retries them.
- The API server maps GitHubRateLimitError to 429 (its GitHubAPIError
handler would otherwise have returned 502).
Also follow-ups from #64: add_assignee now returns the resulting
assignees so R-PR-001 reports an error if GitHub accepted the request
without assigning the author, its docstring no longer overclaims
GitHub's 404 behavior, and README dashes are fixed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BigLep
changed the title
fix: retry GraphQL mutations on GitHub's secondary rate limit
fix: defer rate-limited board writes instead of failing the run
Oct 4, 2026
GitHub reports GraphQL throttling as HTTP 200 with errors of type
RATE_LIMITED, which graphql_query surfaced as a generic GitHubAPIError.
set_field_value_bulk then fell back to per-item writes instead of
stopping and deferring (60 items made 63 requests). Raise
GitHubRateLimitError for that shape too, per review on #65.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GitHub's docs warn that continuing to make requests while rate limited
may get the integration banned. The previous change only stopped inside
one set_field_value_bulk call; the remaining value groups and every
later rule kept querying. A rate limit outside the bulk write also
escaped uncaught, crashing the CLI before it saved the mutation log for
writes that had already landed (R-FC-012 relies on that log).
- github-projects-client: one public is_rate_limited() for every shape
GitHub documents (429; 403 with exhausted quota, Retry-After, or a
rate-limit/abuse message; GraphQL 200 whose errors report it), used
by graphql_query and the server.
- foc-mechanical-rules: build_session mounts a RateLimitBreaker adapter
that refuses to send anything after the first rate-limited response.
Rule.run reports the tripping item and all later items as deferred
and skips the batched flush; runner reports later rules as not run;
a rate limit raised while resolving a bulk write defers that group.
- cli saves the mutation log even if the run raises.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cycle rules and R-PR-010 each had an identical mutate_pending
(group by value, bulk write, defer on rate limit, map results), so
every change had to be made twice. Move it to a BatchedFieldRule base
class keyed by board_field; rules now only select and decide.
Also stop is_rate_limited from decoding every successful response:
only GraphQL 200s can carry a rate limit, so REST pages of board items
passing through the session's breaker are no longer parsed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- RateLimitBreaker raises GitHubRateLimitError on the response that
trips it, so REST helpers (list_items' raise_for_status) no longer
surface a plain HTTPError that crashed the run during select().
- The breaker logs GitHub's message and rate-limit headers once, so the
workflow log shows which limit was hit even though items are deferred.
- set_field_value_bulk's old-value read lets rate limits propagate, so
nothing is written after a throttled read in sessions without the
breaker (e.g. the API server).
- The per-item fallback after a failed batch waits 1s before each
write, per GitHub's guidance for mutative requests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Write-time throttling does not reach this handler: _execute_batch() catches GitHubRateLimitError and returns per-item failures, then server/routes/mutations.py:92-104 returns HTTP 200 and drops the new rate_limited flag. After successful preflight reads, even a throttled first mutation therefore provides no structured throttle or retry/reset metadata. Preserve that metadata in bulk results and handle it in the route: return 429 when all writes are deferred, while retaining successful results for partial completion. Add a route test that throttles the actual write; the current test only mocks the bulk call raising.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rjan90
left a comment
Summary
The
FOC Board Mechanical Rulesworkflow failed twice on 2026-10-03 (run 37081217551, run 37102189769) during the Cycle rollover: R-FC-013 had hundreds of Cycle updates queued, and after roughly 200 successful writes every later write got403 Forbidden. That pattern points to GitHub's secondary rate limit, but the logs never captured the response body, so it isn't confirmed. This PR also makes the next failure explain itself.An earlier version of this PR retried with sleeps inside
graphql_query. That was replaced because the retries stacked withset_field_value_bulk's per-item fallback (one throttled batch could block for hours), blocked API server requests for minutes, and ignored 429s.github-projects-client
graphql_queryraises a typedGitHubRateLimitError(403 with a rate-limit message or exhausted quota, or 429) and includes GitHub's own error message in every HTTP error. It never sleeps or retries.set_field_value_bulkwaits 1s between mutation batches (GitHub's guidance for writes). On a rate limit it skips the per-item fallback, stops sending requests, and marks the remaining itemsrate_limited.GitHubRateLimitError; without a dedicated handler itsGitHubAPIErrorhandler would have returned 502.foc-mechanical-rules
deferred, which doesn't fail the run: the next hourly run retries them.errorstill fails the run, so problems that need a person (like a missing repo permission) stay loud.BatchedFieldRulebase class now owns the batched write (group by value, bulk write, defer on rate limit, map results) that R-FC-012/013/014 and R-PR-010 each duplicated; rules only select and decide.add_assigneereturns the resulting assignees, and R-PR-001 now reports an error if GitHub accepted the request without assigning the author (previously this would have shown as "applied"). Its docstring no longer overclaims GitHub's 404 behavior, and the README documentsdeferred.Follow-ups after review (9742135, 4ef0550, c40e67e)
RATE_LIMITEDerrors) is now detected (thanks @rjan90). Detection lives in one publicis_rate_limited()covering every shape GitHub documents, including a 403 withRetry-Afterand the older "abuse detection" wording.RateLimitBreakerthat sends nothing after the first rate-limited response: remaining items aredeferred, later rules are reportednot run, and the run doesn't fail.Test plan
github-projects-client:uv run pytest -m "not integration"(102 passed), including rate-limit classification, batch pacing, stop-on-rate-limit with no fallback, non-throttle fallback still working, clear-mode batching, and the server's 429 mappingfoc-mechanical-rules:uv run pytest -m "not integration"(74 passed, including the breaker stopping requests, deferral after a trip, later rules not run, and the mutation log surviving a crash), includingdeferredresults not failing the CLI,errorstill failing it, and the silent-non-assignment check.githooks/pre-commitdeferred, and check the logged GitHub message to confirm the rate-limit diagnosis🤖 Generated with Claude Code