Add resilient external attrsvc lifecycle

[namitdhameja]

Summary

• Stable external endpoint: A new Slurm supervisor daemon keeps attrsvc on the same allocated CPU node and restarts it after process crashes, allowing NVRx to continue using the same endpoint.
• Recover from lost requests during GET: For example, attrsvc accepts cycle 2, then restarts and loses its in-memory state. NVRx's subsequent GET receives 404, abandons that cycle, and frees the request slot so a later cycle can still be analyzed.
• Bound GET outages: After three consecutive failed GET attempts, NVRx abandons that cycle instead of polling indefinitely and blocking later attribution requests.
• Avoid a second FT-path delay: When a terminal POST is not accepted, NVRx skips the immediately following optional progressive POST. This can save up to two seconds; terminal POSTs are never skipped.

Testing

• PYTHONPATH=src .../.venv/bin/python -m pytest tests/shared_utils/test_health_check.py tests/attribution/unit/test_attrsvc_restart_agent_http.py services/tests/test_attrsvc_slurm_supervisor.py -q
  • 107 passed, 5 existing warnings, 9 subtests passed
• PYTHONPATH=src .../.venv/bin/python -m pytest tests/attribution/unit/test_client_response.py tests/shared_utils/test_health_check.py -q
  • 115 passed, 4 existing warnings, 13 subtests passed
• Black, isort, Ruff, git diff --check, and shell syntax checks

[greptile-apps]

Confidence Score: 5/5

[High impact] The PR appears safe to merge; no actionable issues remain in the reviewed changes.

Summary

Adds bounded recovery for an external attrsvc and keeps its Slurm allocation alive across process crashes.

• Slurm keeps attrsvc on its allocated node through brief crashes.
• A rejected terminal POST skips the next optional progressive POST.
• NVRx releases lost requests and bounds failed GET polling.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Failed workload cycle] --> B[Terminal POST]
  B --> C{Accepted?}
  C -->|No| D[Leave slot free and skip one progressive POST]
  C -->|Yes| E[Poll GET]
  E --> F{Response}
  F -->|Pending or in flight| G[Reset failure count]
  G --> E
  F -->|Availability failure| H{Three consecutive failures?}
  H -->|No| E
  H -->|Yes| I[Release slot]
  F -->|Lost or rejected request| I
  F -->|Completed| J{STOP?}
  J -->|Yes| K[Latch STOP]
  J -->|No| I

Loading

_{Reviews (13) · Last reviewed commit: "Merge branch 'main' into fix/attrsvc-slu..." · Reviewed by Greptile}