feat: add automated SP dataset cleanup jobs

[beck-8]

https://telemetry.betterstack.com/team/t468215/dashboards/618457/charts/18254157065?rf=now-7d&rt=now
Situation where the testnet dataset is exceeded

Summary

• New global job sp_dataset_pruning: prunes blocked SPs to 0 active
  data sets, and prunes any other SP (trickle-tier or full-rate)
  whose active data-set count exceeds its tier's target by more than
  <NET>_EXCESS_DATASET_BUFFER (default 5) — a safety net against
  unbounded accumulation regardless of cause.
• New global job abandoned_dataset_sweep: cleans up data sets
  belonging to permanently unreachable SPs via PDPVerifier's
  permissionless deleteDataSet path (no signature needed, reclaims
  the cleanup deposit via cleanupPieces), and auto-resolves stuck
  rail settlements via the permissionless settleRail where
  possible — only escalating genuinely stuck cases (validator issues)
  to a human, surfaced via a structured log with a ready-to-use Safe
  Transaction Builder batch.
• New metrics: sp_termination_attempts_total,
  sp_termination_stuck_gauge.
• Runbook: docs/runbooks/wallet-and-session-keys.md (Automated SP
  Cleanup section).

Follow-ups (not in this PR)

• infra: add a BetterStack/Grafana alert rule on
  sp_termination_stuck_gauge.
• Separate investigation: why some SPs (e.g. calibnet 2/4/9, 23)
  over-accumulate active data sets. Confirmed at least one distinct
  mechanism for 23: data_set_lifecycle_check throwaway sets whose
  immediate termination keeps failing while proving stays healthy —
  so abandoned_dataset_sweep's inactivity-based cleanup never
  triggers for them (root cause likely on that SP's own RPC/relay
  path, not a dealbot bug). 2/4/9 haven't been traced yet — could
  be the same termination-failure pattern, a data-set-reuse issue in
  provisionNextMissingDataSet, or something else. This PR's
  excessDatasetBuffer pruning retries provider-relay termination
  daily regardless of cause, which helps if failures are intermittent
  but can't resolve a persistently broken termination path.

Test plan

• pnpm --filter dealbot-backend test — 708 passing
• pnpm --filter dealbot-backend typecheck
• biome check
• Deploy to calibnet first; watch the first abandoned_dataset_sweep
  run (large one-time historical backlog, will take a while and use
  real testnet FIL for gas) and the session key's FIL balance
  (wallet_balance{currency="FIL"})

related #605 #681 #689

[Copilot]

Pull request overview

Adds automated, network-aware cleanup of excess and abandoned SP datasets, including rail settlement recovery and operational monitoring.

Changes:

• Adds scheduled dataset pruning and abandonment sweep jobs.
• Adds cleanup metrics, configuration, and queue integration.
• Documents configuration and operator recovery procedures.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
docs/runbooks/wallet-and-session-keys.md	Adds cleanup runbook.
docs/environment-variables.md	Documents cleanup settings.
apps/backend/.env.example	Adds example variables.
apps/backend/src/config/constants.ts	Defines defaults.
apps/backend/src/config/env.schema.ts	Validates variables.
apps/backend/src/config/loader.ts	Loads cleanup configuration.
apps/backend/src/config/network-fields.ts	Registers network variables.
apps/backend/src/config/types.ts	Adds configuration types.
apps/backend/src/data-set-lifecycle/data-set-lifecycle.service.ts	Exports relay termination helper.
apps/backend/src/database/entities/job-schedule-state.entity.ts	Adds job types.
apps/backend/src/jobs/job-queues.ts	Defines cleanup queues.
apps/backend/src/jobs/jobs.module.ts	Imports cleanup module.
apps/backend/src/jobs/jobs.service.ts	Schedules and runs cleanup jobs.
apps/backend/src/jobs/jobs.service.spec.ts	Updates job-service fixtures.
apps/backend/src/jobs/repositories/job-schedule.repository.ts	Adds queue metric mappings.
apps/backend/src/metrics-prometheus/metrics-prometheus.module.ts	Registers cleanup metrics.
apps/backend/src/sp-cleanup/sp-cleanup.module.ts	Defines cleanup module.
apps/backend/src/sp-cleanup/sp-cleanup.service.ts	Implements pruning and sweeping.
apps/backend/src/sp-cleanup/sp-cleanup.service.spec.ts	Tests cleanup behavior.
apps/backend/src/wallet-sdk/wallet-sdk.service.spec.ts	Updates network fixture.

Suppressed comments (2)

apps/backend/src/sp-cleanup/sp-cleanup.service.ts:343

• Important: verify that the mined deletion transaction succeeded before recording a successful attempt. Viem's waitForTransactionReceipt resolves for receipts whose status is reverted; this path would increment the success metric, log the data set as deleted, and enter cleanup even though no state changed.

      const hash = await writeContract(writeClient, request);
      await waitForTransactionReceipt(writeClient, { hash });
      this.recordAttempt(network, "abandonment", "success");

apps/backend/src/sp-cleanup/sp-cleanup.service.ts:393

• Blocker: make piece-cleanup failures verifiable and retriable. A mined reverted receipt is currently treated as success here, while the catch below converts every RPC, submission, or receipt failure into “cleanup not needed” and returns. After deleteDataSet has entered cleanup mode, later sweeps have no path that resumes cleanupPieces, so remaining pieces and the FIL deposit can be stranded permanently. Check receipt.status, suppress only the specifically decoded already-finalized/not-in-cleanup condition, and ensure cleanup-mode data sets are rediscovered and resumed after transient failures.

        const hash = await writeContract(writeClient, request);
        await waitForTransactionReceipt(writeClient, { hash });

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Cleanup recovery, rail finalization, provider coverage, and timeout enforcement contain correctness gaps that could leave resources permanently stranded.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

apps/backend/src/sp-cleanup/sp-cleanup.service.ts:552

• Important: this is not the Safe Transaction Builder checksum algorithm, so every generated “ready-to-use” batch is flagged as modified on import. Safe canonicalizes recursively with sorted keys, removes the checksum, and sets meta.name to null before hashing; hashing plain JSON.stringify(batch) produces a different value. Use Safe’s canonical checksum implementation (or intentionally omit the optional checksum) and add a compatibility test against a known batch.

docs/environment-variables.md:709

• Important: this description contradicts the implementation and runbook by saying stuck rails can only be surfaced to a human; the job first resolves them automatically with permissionless settleRail and escalates only reverts. Update this operational documentation so responders know which cases require the Safe batch.

**Role**: How often the `abandoned_dataset_sweep` job runs for this network. This global job scans dealbot's
entire wallet (not scoped to the blocklist) for data sets outside the PDPVerifier activity window and deletes
them directly and permissionlessly (no signature needed — see the runbook), and for terminated data sets whose
rail settlement is stuck past `endEpoch`, which it can only ever surface to a human operator via a structured
log (`stuck_terminations_detected`).

• Files reviewed: 20/20 changed files
• Comments generated: 5
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Dataset selection, post-delete recovery, timeout enforcement, and reverted-settlement classification have unresolved correctness risks.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

apps/backend/src/sp-cleanup/sp-cleanup.service.ts:408

• Blocker: Make post-delete piece cleanup recoverable. These lines confirm that a successful deleteDataSet removes the ID from the only discovery path, yet the method can subsequently exit on the 20-minute abort, a process restart, five transient RPC failures, or the iteration cap. Any such exit permanently strands the remaining pieces and cleanup deposit, which is especially likely during the documented large historical sweep. Persist or derive cleanup-mode data-set IDs independently of listDataSets and retry them across runs before considering deletion complete.

    // `deleteDataSet` already removed this data set from FWSS's `clientDataSets`, so a future
    // sweep's `listDataSets` can never rediscover it to retry a failed batch — every reasonable
    // effort has to happen right here, in this one call.
    const MAX_CONSECUTIVE_FAILURES = 5;

• Files reviewed: 20/20 changed files
• Comments generated: 3
• Review effort level: Balanced

[BigLep]

@beck-8 : thanks for posting.

I would like to take a look (I've assigned myself as a reviewer) but don't block on me.
I'm also thinking to put @Kubuxu rather than @SgtPooki given Kuba is on the Storage Service workstream and Russell is not (but others feel free to step in here if you disagree.)
That said, Kuba is likely backlogged on reviews with helping looking at builtin-actors for nv29 this week.

[Kubuxu]

I can't meaningfully review this PR this week. Even after that, I have almost no knowledge of the dealbot codebase. If I review this PR, it is going to be strongly LLM-guided.

[silent-cipher]

Also another nit: can we use dataSet instead of dataset?

[beck-8]

At this point, I’m feeling a bit overwhelmed—the number of changes has far exceeded my expectations, and I can’t guarantee that the AI will execute exactly as I envision. From what I’ve seen so far, there are still many omissions. When we’re cleaning up, would it be better to add a persistent data table to track these states? That way, we could reduce RPC calls, continue tracking even if a task fails, and handle edge cases during cleanup. I’d like to ask someone else to take over this task or replace it outright.

[silent-cipher]

I asked the agent to prepare a Q&A from the review discussion, and I'm pasting it here to make reasoning easier for everyone:

SP cleanup jobs: consolidated rationale

The conclusions below were checked against the deployed contract behavior and the live calibration wallet—not inferred only from this repository.

Summary

• Keep sp_data_set_pruning and abandoned_data_set_sweep as separate global jobs.
• Pruning handles cooperative SPs through the provider relay.
• The sweep handles abandoned datasets and delayed rail settlement through permissionless on-chain calls.
• Neither job belongs on SP_WORK_QUEUE.
• The sweep addresses an existing calibration backlog, not a theoretical problem.
• Incomplete cleanupPieces work is acceptable because stopping proofs and releasing dealbot’s lockup happen earlier.

Why two separate global jobs?

Job	Purpose	Authorization	Why global
sp_data_set_pruning	Remove blocked or excess datasets	Requires a reachable, cooperative SP through the relay	Dataset listing is wallet-wide. Running per SP would repeatedly fetch the same wallet data.
abandoned_data_set_sweep	Delete abandoned datasets and settle terminated rails	Uses permissionless on-chain paths	It must include datasets whose SP has disappeared from the registry, and it never contacts the SP server.

The sweep is not merely a fallback for pruning. Most terminated datasets come from the existing lifecycle-check job, while abandoned datasets can belong to SPs that pruning would never target.

Why pruning does not use SP_WORK_QUEUE

The existing per-SP queue is already close to full worst-case occupancy: approximately 3,492.5 seconds per hour for a full-rate SP.

Adding another consumer would most affect busy or degraded SPs—the same ones most likely to need pruning. Blocked SPs already have no collision risk because they are excluded from normal SP scheduling. For non-blocked SPs, pruning follows the existing maintenance-window deferral and performs only brief relay requests.

Contract behavior and ownership of funds

The deleteDataSet path does not have the session-key problem seen in #546:

• FWSS.terminateService requires the caller to be the payer or SP, so dealbot’s session key cannot call it directly.
• PDPVerifier.deleteDataSet requires the SP only during the approximately 30-day activity window. After that, it is permissionless.
• FWSS independently checks the same inactivity condition when processing dataSetDeleted.

One correction from the earlier discussion: the PDP cleanup deposit is funded by the SP, not dealbot. Whoever completes cleanup receives that deposit. It is therefore a keeper bounty, not dealbot recovering its own funds.

The payment-rail lockup is different: that is dealbot’s money. Settling the rail releases funds belonging to dealbot.

Why Curio cleanup does not make the sweep redundant

Curio cleanup helps, but dealbot should not depend on it exclusively:

1. A dead SP cannot submit cleanup transactions.
2. SP cleanup recovers the SP-funded deposit, not dealbot’s funds.
3. Curio cleanup is rate-limited, so rail settlement may be delayed indefinitely from dealbot’s perspective.

A cooperative SP may eventually settle the rail itself. The reason for dealbot to do it is control over when its own lockup is released—not because settlement is otherwise impossible.

Current calibration evidence

At approximately block 4,077,068 on 2026-09-17:

• 22 datasets were active but never terminated.
• 15 of those were already outside the inactivity window, some stale for 101–233 days.
• 49 terminated datasets were past their payment end epoch.
• 35 had been cleaned up.
• 14 remained stuck, including 10 for roughly 228 days.

That is a 29% stuck rate among eligible terminated datasets. The sweep is addressing a real backlog. This check has not yet been repeated against mainnet.

Accepted limitation

If the job times out after deleteDataSet succeeds but before cleanupPieces finishes:

• proof obligations have already stopped;
• dataSetDeleted has already abandoned the rails and released dealbot’s lockup;
• only the SP-funded cleanup bounty remains unclaimed.

That outcome satisfies dealbot’s goals, so no additional recovery mechanism is planned.

[silent-cipher]

Found a throughput problem with both cleanup jobs: they process data sets strictly in sequence, and each termination/deletion needs an on-chain confirmation (~30s block time). With a 20-minute job window, that caps us at roughly 40 data sets cleaned per run in ideal case. On calibration, some SPs have ~500 data sets to terminate, at this rate that's weeks to clear a single SP's backlog, and other SPs would sit starved behind it in the meantime.

Holding off on merging until this is resolved.