feat(identity): activate signature observability in drivers, with metrics and docs (3/3 of #2084)

[Effi-S]

Split 3/3 of #2084 - the feature goes live

• token/services/identity/metrics.go (+ tests, golden, reference_test): the Prometheus sink implementing Observer + LevelGauge. Kept together because the metricsdoc self-test binds metrics.go to the golden, the metrics doc, and the driver wiring sites.
• fabtoken / zkatdlog ws.go: build the sigpolicy.Stack, install the observer on the identity provider and the deserializer, return the stack with a transferred-guarded Stop() lifecycle.
• fabtoken / zkatdlog driver.go: hand the validator its own un-observed NewDeserializer so ledger-named token owners can never feed the escalator (the core security fix), then install the stack on the client-facing service.
• ttx/auditor.go: drop the now-dead SignatureThrottled branch.
• docs: signature_observability.md plus configuration/metrics/grafana updates and the metrics golden.

[AkramBitar]

Review: signature observability activation

Read every hunk with surrounding context and verified the branch builds and tests: go vet ./token/... clean; tests pass for metricsdoc, identity, sigpolicy, throttle, sigobserve and both driver packages; integration/ and the nested x/token/services/network/evm module build. I also checked the new goleak test has teeth — removing defer sigStack.Stop() in ws.go fails it with both Escalator.evictLoop and BucketSet.evictLoop leaked.

No compile- or crash-level bug. The transferred ownership pattern, the un-observed validator deserializer (the core security fix) and the nil guards in Metrics.Observe all check out.

9 findings inline: 4 medium, 5 low. The two I would not merge without a decision on are the mode: off documentation mismatch and the missing PrincipalKeyResolver on the Idemix driver — together they mean the knob operators are told to use does less than advertised at both ends of the range. The teardown-ordering one is a real fail-open, though only on an already-failing update path.

The redundancy findings on metrics.go are judgement calls, not defects — flagging them because two of the three new families are derivable from counters this PR leaves in place.

[AkramBitar]

medium — mode: off is documented here (and at docs/security/signature_observability.md:217) as "nothing metered, nothing denied", but only the denial half is true.

sigpolicy.New appends the metrics and audit observers unconditionally and gates only the escalator on cfg.Enabled(). Both drivers always pass a non-nil logger and a non-nil identity.Metrics (ws.go:59, ws.go:107), so instantiating the stack with Mode: off yields a sigobserve.multiObserver, not Nop (verified by probe).

Consequence: an operator who sets off specifically to shed overhead on a latency-sensitive node still pays, per signature operation, an Identity.UniqueID() hash, two time.Now() calls, 2-3 Prometheus family updates, an audit-logger level probe and an instrumentedVerifier allocation. The documented zero-cost d.observer == Nop fast path in common.Deserializer.getVerifier is never taken in production.

Either have off drop the sinks, or reword both docs to "policy off; instrumentation unaffected".

[AkramBitar]

medium (redundancy) — identity_signer_cache_lookups_total carries no information that is not already exported.

It has exactly one producer, Timer.DoneResolution, called only at provider.go:410 — the same statement that already does SignerResolutions.With("outcome", path).Add(1). Since DoneResolution sets CacheHit = err == nil && path == PathCache:

• {result="hit"} is identical to identity_signer_resolutions_total{outcome="cache"}
• {result="miss"} is the sum of the other outcomes plus errors

So this adds a metric family, a golden-file entry, a docs row and per-call work that a recording rule over the existing counter would cover.

[AkramBitar]

low (redundancy) — identity_signature_operation_duration_seconds{op="get_signer"} is sum without(path) of the pre-existing identity_get_signer_duration_seconds{path}, and both are observed from the same measurement at provider.go:406-410.

Each declares 13 explicit buckets plus native-histogram options, so GetSigner — on the endorsement hot path — now pays two full histogram observations for one timing. Worth dropping one, or deriving the aggregate in Prometheus.

[AkramBitar]

medium — activating the stack here means the gate runs on the Idemix driver, whose owner identities are pseudonyms, but no PrincipalKeyResolver is installed anywhere: WithPrincipalKeyResolver has zero call sites outside its own declaration in token/sig.go.

SignatureService.allow therefore keys on id.UniqueID(). A caller presenting a fresh nym per OwnerVerifier/GetAuditInfo call gets a brand-new full bucket every time and is never denied — exactly the failure mode token/sig.go:43-56 warns about.

This matters most for an operator following "Rolling out enforcement" (docs/security/signature_observability.md:235): switching to enforce on this driver looks like it has throttling but does not. Either wire a resolver or state the limitation in the rollout guide and in docs/configuration.md.

[AkramBitar]

medium — teardown ordering makes enforce fail open after a failed public-parameters update.

common.Service.Done() (token/core/common/tms.go:192, outside this diff) stops the signature instrumentation before the fallible walletService.Done(). If the wallet service's Done() fails — wallet/service.go:250 joins registry errors — then TMSProvider.Update (token/core/tms.go:182) returns that error and leaves the old service in m.services[key], and token.ManagementServiceProvider.Update (token/provider.go:131) returns before delete(p.services, key).

Result: the cached TMS keeps serving requests with an already-stopped — and therefore inert — gate, so enforcement silently stops denying until restart. Separately, the freshly built newService is dropped without Stop(), leaking its two eviction goroutines plus escalator/bucket state; before this PR a discarded token service held no goroutines.

Stopping the instrumentation only after the wallet teardown succeeds (or on the error path, stopping the new service) would close both.

[AkramBitar]

low — the comment above correctly describes this as deliberate, so this is about the cost it leaves behind rather than a correctness bug.

On the wallet-only path (same at token/core/zkatdlog/nogh/v1/driver/ws.go:70) the stack is stopped but identityProvider.SetObserver / deserializer.SetObserver stay wired, so the observer is a multiObserver, not Nop. Every resolution hashes the identity and allocates an instrumentedVerifier for a policy that is inert, and since both factories pass &disabled.Provider{} the metrics go nowhere — the only live sink is the audit logger. Passing nil sinks to sigpolicy.New on this path would collapse the chain to Nop and keep the documented fast path.

One side effect worth a look regardless: Stack.Stop() zeroes identity_throttled_principals. Harmless with disabled.Provider, but zkatdlog's BaseWalletServiceFactory.NewWalletService is exported and promoted onto *Driver, so a caller passing a real TMS-scoped provider would clear a live TMS's gauge.

[AkramBitar]

low — this paragraph tells callers of AuditorVerifier to be ready for token.SignatureThrottled "from a gated verifier inside the deserializer", which contradicts line 42 of this same file ("the gate is consulted at exactly one place"), token/sig.go, and the comment this PR adds at token/services/ttx/auditor.go:304 ("this can never be a token.SignatureThrottled denial").

common.Deserializer holds an Observer, never a Gate, so the code comment is right and this paragraph is wrong. A reader following it adds unreachable error handling.

[AkramBitar]

low — escalation is listed among the op label values for identity_signature_operations_total, but Metrics.Observe returns early for OpEscalation (metrics.go:139), so identity_signature_operations_total{op="escalation"} is never produced — as this PR's own test asserts ("policy state must not be counted as a service call").

An operator writing sum by (op) queries from this list will look for a series that does not exist.

[AkramBitar]

low (scope) — renaming the exported PayerAccessTokenExp to PayerAccessTokenExpiry and regrouping this const block is unrelated to signature observability.

Nothing breaks — the single in-repo user is updated — but it is a gratuitous exported-identifier change in a PR that is otherwise about the identity/driver layer, and it enlarges the diff for reviewers. Better as its own commit or PR.