Skip to content

fix(ateapi): add store fallback in ActorIdentity to resolve test flakiness - #1113

Open
shrutiyam-glitch wants to merge 2 commits into
agent-substrate:mainfrom
shrutiyam-glitch:fix-flaky-2
Open

fix(ateapi): add store fallback in ActorIdentity to resolve test flakiness#1113
shrutiyam-glitch wants to merge 2 commits into
agent-substrate:mainfrom
shrutiyam-glitch:fix-flaky-2

Conversation

@shrutiyam-glitch

@shrutiyam-glitch shrutiyam-glitch commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Fix TestDurableDirLifecycle flaky test
https://github.com/agent-substrate/substrate/actions/runs/32416349094/job/96578260374

Cause:

  • In the specific test (_suspend_from_PAUSED), the actor is suspended (which strips its worker assignment, returning the worker to the pool) and then immediately resumed.
  • When the test calls ResumeActor, the control plane selects a new worker, writes the assignment to the Kubernetes database, and synchronously makes a direct gRPC Restore() call to the atelet on that new worker.
  • The atelet boots the actor and atunnel. atunnel then immediately asks the ateapi (via the ActorIdentity service) to mint an actor certificate so it can connect to the network.

Race: The ActorIdentity service verifies that the worker is actually assigned to the actor by checking its internal workercache.Cache. However, this cache is powered by Kubernetes Informers, which receive updates asynchronously. Because ResumeActor was so fast, the cache hasn't processed the new worker assignment yet.
The ActorIdentity service looks at the stale cache, sees the worker is unassigned, and hard-rejects the certificate minting with caller is not permitted to mint credentials for this actor.

I faced this issue too when working on the postgresql updateWorker optimization (#934)

Fix: Updated authorizeActor inside cmd/ateapi/internal/actoridentity/actoridentity.go.
Now, if authorizeActor fails an authorization check (like discovering a missing or mismatched worker assignment), it will automatically do a one-time "live fetch" bypass (s.store.GetWorker()) directly against the primary store to bypass the lagging informer cache.

  • Tests pass
  • Appropriate changes to documentation are included in the PR

@BenTheElder

Copy link
Copy Markdown
Collaborator

Now, if authorizeActor fails an authorization check (like discovering a missing or mismatched worker assignment), it will automatically do a one-time "live fetch" bypass (s.store.GetWorker()) directly against the primary store to bypass the lagging informer cache.

If we're failing through commonly ... maybe we shouldn't even be using it?

I haven't had a chance to dig yet, but IIRC Zoe Zhao (@zoez7) was moving actor out of CRD => substrate API?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants