[Peeja]

Written by Claude.

Every alert rule carries a severity of critical, warning or info, but nothing said what they mean. docs/observability.md only described how routing uses them. This PR adds the definitions after the paragraph on alerts with no stage:

• critical: customers are affected now, or something will be lost for good before the next day unless someone acts: a proof set Curio is about to give up on, failures counting toward that, or work Piri is about to stop retrying. Pages in production, and when the alert has no stage.
• warning: needs action the same or the next working day. That includes a loss that grows while it lasts but stops once the cause is fixed, such as a missed proving period's payment: mainnet's period is 24 hours, so the next working day loses about one period. Weekends are a known gap (FIL-1509). Notifies.
• info: a trend to look at; nobody needs to act on it right away. IRM's routes can treat it more quietly than a warning.

It also gives the test for making a rule critical: a person paged now can do something about it, and waiting until the next day would cost something that can't be recovered.

The severity is judged by the production impact, and the label is the same on every stage. Staging doesn't page because of the routing decided in FIL-1164, not because of the label.

Where a missed proving period falls

A missed proving period pays nothing for that period on that data set. That is FWSS's validatePayment ("Faulted: Deadline has passed with no proof … (zero payment)") in fil-forge/filecoin-services at 36ebe44, service_contracts/src/FilecoinWarmStorageService.sol lines 1636–1739, the version pinned in Piri's go.mod. There is no slashing.

An earlier version of this PR counted that payment as critical. Following review (hannahhoward, and bajtos on Slack), it's now a warning. The loss stops once the cause is fixed, and mainnet's 24 h period leaves time to fix it the next day. What still pages is a loss that waiting would make permanent: Curio giving up on a proof set after five rejected transactions, and Piri giving up on pieces it was adding (FIL-1508). fil-forge/infra-central#249 applies this test to each Piri rule.

A warning that fires on Friday evening waits until Monday, about 60 h and 2–3 proving periods. That gap is FIL-1509, not this PR.

Not changed

• No rule's severity changes; this PR is docs only.

Checks

• make check passes.
• git diff --check is clean.

✴️

🤖 Generated with Claude Code

https://claude.ai/code/session_01JCF8gyb8jNRmHmWqQmVhRJ

---

Generated by Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Peeja]

Claude reviewer says… (1st round, eb1a545)

Approve with nits. I found nothing blocking or should-fix.

Verified

• The diff only adds docs/observability.md:87-101, after the paragraph on alerts with no stage. No other file changes.
• The text matches routing.tf: one contact point, grouped by stage and severity, with IRM picking the escalation. It also matches the doc's own "a production critical alert pages and everything else notifies".
• "One severity for every stage" holds: every severity in alerts.tf is a literal.
• The text is consistent with the comments in alerts.tf on the Piri rules (L1692-1694), the outages group (L2636-2637) and the info trace-volume rule (L2882-2884).
• In FWSS validatePayment at 36ebe44, a period whose deadline passed unproven is Faulted and pays zero (L1698). Termination only happens when the payer or provider calls it (L1067).
• 36ebe4467390 matches Piri's go.mod.
• make check passes and git diff --check origin/main... is clean.

Findings

1. nit, PR body. The FWSS link has backticks inside its target, so it doesn't render as a link.
2. nit, PR body. "Pays nothing … on every data set the node serves" overstates it. Proving periods are per data set: a node-wide failure, such as a stale head, misses them all, but one failing proof set loses only its own.
3. nit, docs/observability.md:99-101. "Otherwise it is a warning" leaves out info, and "A new rule" suggests existing rules are exempt from the test.
4. nit, docs/observability.md:94. "Pages in production" leaves out the case the doc covers a few lines above: a critical alert with no stage pages too.

Observations on existing rules (not for this PR)

• "Appliance container telemetry has stopped" (critical). This is a monitoring blind spot rather than customer impact, so it reads like a warning under the new text.
• "Appliance has stopped reporting" (critical). Its own comment says the common cause is a stopped reconcile timer. That means stale deploys, not customer impact.
• "Piri proof set is unrecoverable" (critical). Curio has already given up on the set, so it's unclear what someone paged at night can do that couldn't wait until morning. This is speculation.

✴️

---

Generated by Claude Code

[Peeja]

Claude author says…

All four findings are fixed in fbce363 and in the PR body.

1. Broken link. The FWSS link in the body is fixed and renders.
2. Per data set. The body now says a missed period pays nothing on that data set, and that a node-wide failure, such as a stale chain head, misses every data set.
3. The test. It now reads "A rule is critical only when … Otherwise it is a warning, or info if nobody needs to act on it."
4. No stage. The critical bullet now reads "Pages in production, and when the alert has no stage."

I also corrected two smaller things in the body: where the text sits, and the info bullet's IRM sentence.

The three observations about existing rules are left for Petra to decide. Changing a rule's severity is outside this docs-only PR.

make check and git diff --check pass. This round found only nits, so there won't be another.

✴️

---

Generated by Claude Code

[bajtos]

LGTM. I would like @hannahhoward and @alanshaw to review this proposal too.

[Peeja]

Got the verbal ✅ from @hannahhoward, with the caveat that the proving period alert should be a warning (as @bajtos said in Slack). Now that I understand a little more, I'm on board with that. The mainnet proving period is 24h, we have time to handle it without a page in the middle of the night. PR for that incoming.

E: Actually, that can just go in #249.

[Peeja]

Claude reviewer says… (2nd round, 0c51309)

Approve with nits. Nothing blocking or should-fix. This round covered #248, #249 and fil-forge/infra-nodes#178 together.

Verified

• The new critical/warning wording agrees with Make alert severities match their definitions #249's rule severities and with the runbook's timeframes.
• make check and git diff --check pass. Define alert severities in the observability docs #248 and Make alert severities match their definitions #249 merge cleanly together.

Findings

1. nit, PR body: the FWSS link has backticks inside its target again, so it doesn't render.
2. nit, docs/observability.md:91-93: "a proof set Curio gives up on" as a critical example can be read as making "Piri proof set is unrecoverable" critical, while Make alert severities match their definitions #249 makes it a warning. Suggest "is about to give up on".
3. nit, docs/observability.md:104-106: the closing test still says "working hours", while the bullet says "before the next day".

✴️

---

Generated by Claude Code

[Peeja]

Claude author says…

Fixed in faa3f97.

1. Link: the session's GitHub proxy rewrites links to fil-forge/filecoin-services, a repository outside this session's scope, so the body cites the file as plain text instead: 36ebe44, service_contracts/src/FilecoinWarmStorageService.sol lines 1636–1739.
2. Examples: now "a proof set Curio is about to give up on" and "work Piri is about to stop retrying".
3. Closing test: now "waiting until the next day would cost something that can't be recovered".

Checks: make check, git diff --check.

This round found only nits, so there won't be another.

✴️

---

Generated by Claude Code

[Peeja]

Claude says… plan-dev-apps failed at tofu plan on b6e7e9c, the merge of main into this branch. This PR changes only docs/observability.md, so nothing in it reaches that plan. The same job passed on #249, #228 and #253, which merged the same main at the same time. I can't read the job log from this session, so I don't know the cause; a state lock or an AWS API error is my guess. Re-running the job once.

✴️

---

Generated by Claude Code