Slow-but-successful checks are recorded as permanent failures: track a degraded outcome class

[SgtPooki]

Problem

A check that completes successfully but slower than our window is recorded identically to one that never worked. Deal checks have a single hard cutoff (dealJobTimeoutSeconds, default 360s in config/constants.ts). When the window expires, classifyFailureStatus emits failure.timedout and the deal row is persisted with status = failed (deal.service.ts#L664). Nothing revisits that row when the addPieces message lands on chain later, so an eventually-successful deal stays a permanent failure in the DB and in every downstream metric.

The 2026-07-20/21 mainnet congestion made this concrete (internal discussion: Slack thread): elevated base fee pushed addPieces confirmations well past the deal window while the data itself stored and retrieved fine, and dealbot recorded deal failures. Those false negatives are indistinguishable from an SP that lost the data.

What this tracks

1. An outcome class between success and failure: "worked, but outside our time requirement" (e.g. a pending_confirm-style state; naming open). Metrics and dashboards need to separate "SP broken" from "SP or chain slow".
2. Headroom between the optimal window and a longer functional-but-degraded window, instead of one binary cutoff. Retrieval check silently records failure.timedout when outer job timeout preempts inner IPNI timeout #540 documented how a single cutoff turns a continuous latency distribution into cliff-shaped failure rates.
3. Reconciling failed rows when the deal later confirms on chain (overlaps database synchronization with chainstate #465).

Related

• Retrieval check silently records failure.timedout when outer job timeout preempts inner IPNI timeout #540 (silent failure.timedout, binary-cutoff artifact), RPC timeout mismatch: dealbot (10s) gives up before eRPC (30s) can fail over #603 (slow RPC recorded as hard deal failures), lower timeouts for deals/retrievals #267 (timeout tuning), database synchronization with chainstate #465 (DB vs chainstate sync)
• pdp: eth sender flattens EIP-1559 transactions to legacy at nonce assignment - every message pays its full fee cap and strands on any baseFee rise filecoin-project/curio#1354 (zero-headroom message caps stranding under high base fee), FilOzone/infra#325

[beck-8]

Is this just dealbot making some changes?
My understanding is that dealbot is equivalent to a real customer. We may need to modify the sdk to change the timeout into a pending state. Should all users benefit?

upload(waiting confirm) -> normal -> Original path
upload(waiting confirm) -> Timeout 5 minutes -> return result{status:pending }
upload(waiting confirm= false) -> return result{status:pending }

[SgtPooki]

@beck-8 this is just an idea, but I think we need some way to discern whether dealbot checks are just missing SLA/KPI, or are delayed because SPs/chain-nodes are processing work too slowly.

currently there are some cases where a "Data storage check" shows as failure at different states, even though if we look at the transaction later, it appears as successful. which means the data storage check failed time expectations, but succeeded eventually. I think this could be a worthwhile data-point, but I don't think the visibility into this should be our priority. and this may all just go away once all SPs are succeeding within timeframes more frequently.

[BigLep]

The issue makes sense to me to make data storage check less binary.

I think we ultimate want these metrics (using the status names defined here):

• data storage ideal:
  • this is what we have today
  • upload, onchain, discoverability, and retrieval status all pass within the allottted short 5m window
  • we surface the existing dataStorageStatus metric, but it isn't part of approval criteria
• data storage minimum:
  • upload, discoverability, and retrieval status all pass within the allottted short 5m window
  • onchain doesn't have to pass
  • this is part of approval criteria
  • this is a new metric like dataStorageMinimumStatus.
• data storage deferred
  • onchain status must be successful within 12 hours
  • this is part of approval criteria
  • this is a new metric like dataStorageDeferredStatus

I'm expecting work in these areas:

• Adjusting the existing data-storage check
• Adding a followup check for the "data storage deferred" check
• Adjustments to https://github.com/FilOzone/dealbot/blob/main/docs/checks/data-storage.md
• Adjustments to https://github.com/FilOzone/dealbot/blob/main/docs/checks/production-configuration-and-approval-methodology.md
• Adjustments/additions to https://github.com/FilOzone/dealbot/blob/main/docs/checks/events-and-metrics.md
• Adjustments to https://probelab.io/filecoin/foc/. We want to expose the "data storage ideal" metric value but denote the approval around "data storage minimum" and "data storage deferred".

Note: the naming here is a bit tricky. These are the names coming to mind to me after a few minutes of thinking, but I'm not sold on them.

[BigLep]

My response above is looking at things purely from the dealbot perspective. If there are changes that should be made to the Synapse SDK or filecoin-pin, please share. Dealbot is supposed to represent a normal customer experience and their changes we should make in our client tools, lets surface.

[BigLep]

Before implementation on this task is started, there should be a quick design review describing:

1. Any changes to the existing data storage check
2. If there is any newcheck being added
3. Metrics that will be emitted

[silent-cipher]

Pushing back on this from a client's perspective.

Dealbot is supposed to represent a client. The chain slowness and delay in add pieces doesn't guarantee the message will land on chain eventually. PDP proving obligations attach to pieces that are part of a dataset's on-chain piece list. If add pieces hasn't landed on chain, the piece isn't covered by the proving/fault mechanism. Nothing on chain is forcing that SP to keep serving it. So uploaded, indexed, retrievable right now without confirmation isn't a weaker deal - it's not really a deal yet. It's just "the SP happens to still have the bytes at this moment."

So I don't think "minimum" or "deferred" should count as any kind of success for approval. A real client isn't going to wait 12 hours without a guarantee, and we shouldn't either.

[TippyFlitsUK]

Operator data point from the SP side, in case it's useful. We hit this class today on calibnet, and the driver was different to the July mainnet case in the issue body.

ezpdpz-calib2 (calibnet provider 2, stock Curio 1.28.4) took ten deal failures between 00:37 and 16:32 UTC today and was still going at 17:25. Seven were Deal job timeout (360s). On those, every step passed except pieces_confirmed: storage, indexing, IPNI verification and retrieval all completed inside about 65 seconds.

What moved was addPieces inclusion latency. It ran at 16 to 40 seconds all morning, stepped to 200 to 440 seconds at 14:09 UTC and had not recovered by 17:25. Sends from the node were steady throughout, no send errors, no task backlog.

Two things here that may bear on the discussion above.

Congestion wasn't the cause this time. Calibnet base fee was at its floor of 100 attoFIL and tipsets were 18 to 70% full. The queue was ours. The node has 1,039 datasets proving on a 240 epoch period, so it emits around 1,050 messages an hour from one sender wallet, arriving in bursts of 29 to 70 a minute against an achieved inclusion of roughly 10 to 13 messages per tipset. That scales with dataset count, so it should be expected to grow as SPs accumulate datasets.

The messages did land, and proving cover held. Every addPieces confirmed. Proving period gaps stayed at 7,198 seconds with a worst case of 7,680 against a 7,200 second target, and the node took 2 faults across roughly 12,470 proving periods today. In this case the pieces were a few minutes late into the proving mechanism rather than outside it.

Possibly relevant to #606: the other three failures today weren't timeouts at all, they were Request exceeds defined limit from eRPC on FWSS eth_calls, two of them on getAllDataSetMetadata. Same node, same day. Since that path is O(active data sets), dataset growth looks like it drives both symptoms.

Happy to pull more detail off the box if any of it helps.

[snadrus]

I see many opportunities for fix here.

1. Quick: We should prefer proving datasets that did not show successful last time (Curio change, easy to roll-out quickly).
2. Explore "deadline shifting" to keep message count under some pre-defined soft limit.
3. Investigate Lotus' "Request exceeds defined limit"
4. Seriously investigate Snark options for PDP to increase scale in chain & SP-dataset count.

[snadrus]

@TippyFlitsUK is this still a problem?
@ZenGround0 This has a few weak mitigations before we need contract changes or proving re-design. Lets discuss how to approach this next meeting.

[TippyFlitsUK]

Generally, I would say yes.