DEP: Standalone EPP observability  #15058

Activity

1. flpanbin

   added

   dep:draftDEP in draft statusDEP in draft status

   on Sep 18, 2026
2. flpanbin

   changed the title [-]DEP:[/-] [+]DEP: Standalone EPP observability [/+] on Sep 18, 2026
3. linear-code

   added

   metrics

   routerRelates to routing, KV-aware routing, etc.Relates to routing, KV-aware routing, etc.

   observabilityRelated to metrics, tracing, loggingRelated to metrics, tracing, logging

   enhancementNew feature or requestNew feature or request

   language::rustIssues/PRs that reference Rust codeIssues/PRs that reference Rust code

   deployment::k8sRelates to dynamo deployment in kubernetesRelates to dynamo deployment in kubernetes

   backend::sglangRelates to the sglang backendRelates to the sglang backend

   backend::trtllmRelates to the trtllm backendRelates to the trtllm backend

   eppCODEOWNER area -> @ai-dynamo/dynamo-epp-codeownersCODEOWNER area -> @ai-dynamo/dynamo-epp-codeowners

   backend::vllmRelates to the vllm backendRelates to the vllm backend

   on Sep 18, 2026
4. 0z5a commented on Sep 18, 2026

   

   0z5a

   on Sep 18, 2026

   Last edited by 0z5a

   More actions

   I feel free to take any task from this. ^^
5. flpanbin

   changed the title [-]DEP: Standalone EPP observability [/-] [+][WIP] DEP: Standalone EPP observability [/+] on Sep 18, 2026
6. tryambak2019 commented on Sep 21, 2026

   

   tryambak2019

   on Sep 21, 2026

   More actions

   @flpanbin I checked the current draft against @0z5a’s #15056. That PR already covers request totals/duration, first-response timing, stream accounting, etc.
   Once the metric contract is settled, I can take one bounded non-overlapping slice-preferably JSONL logging/config parity, or otherwise picks_inflight with the catalog and availability tests. Please tag me when you split the implementation.
7. tmonty12 commented on Sep 22, 2026

   

   tmonty12

   on Sep 22, 2026

   Contributor

   More actions

   Hey @flpanbin, great proposal. A few thoughts:

   Exposing dynamo_component_router_* on the EPP port

   The RouterRequestMetrics struct in lib/llm/src/kv_router/metrics.rs already tracks request count, TTFT, ITL, ISL, OSL, KV hit rate, and overlap metrics — exactly the series this DEP re-proposes under dynamo_epp_*. Right now these are gated behind from_component() which requires a DistributedRuntime, which standalone EPP does not have.

   The fix is a second constructor from_registry(registry, prefix, const_labels) that takes a plain prometheus::Registry and registers the same series with no DRT dependency. The standalone EPP passes in its existing private registry (already serving dynamo_epp_cached_tokens on port 9090) with appropriate const labels.

   On naming: rather than rename these to dynamo_epp_*, I would suggest keeping them as dynamo_component_router_* even on the EPP port. This keeps naming consistent with the DRT/frontend mode (which keeps dynamo_component_router_* on port 8000), avoids a parallel renamed set, and makes it clear the same router logic is running regardless of deployment mode.

   Metrics to remove from the DEP (covered by dynamo_component_router_*):

   Note: dynamo_epp_requests_total and dynamo_epp_time_to_first_token_seconds should stay as EPP owned series. The EPP request total counts at the ext_proc boundary and includes requests that fail before reaching the SelectionService (tokenization failure, semaphore rejection, etc.) — the router counter only increments after admission. Similarly, EPP TTFT is service level (ext_proc receipt to first token back) while the router TTFT starts after tokenization and selection. Both are useful but they measure different things. ITL and OSL are the same in both: they come from the same token stream and response usage field.

   Remove	Covered by
   dynamo_epp_request_input_tokens	dynamo_component_router_input_sequence_tokens
   dynamo_epp_request_output_tokens	dynamo_component_router_output_sequence_tokens

   ITL, KV hit rate, shared cache hit rate, and non-max-overlap selections also come for free.

   Metrics to add (not covered by router metrics, genuinely EPP owned):

   Metric	Type	Notes
   dynamo_epp_tokenizer_duration_seconds	Histogram	EPP owned tokenization step; distinguishes slow tokenization from slow selection
   dynamo_epp_selection_duration_seconds	Histogram	Time inside SelectionService.pick(). Equivalent of dynamo_routing_overhead_total which is frontend only today
   dynamo_epp_selection_errors_total	Counter	Selection failures by reason: no_workers, queue_full, timeout, capacity_exceeded. Needed to diagnose 503s without log parsing
   dynamo_epp_workers_registered	Gauge	Live worker count in the SelectionService. No analog on EPP port today
   dynamo_epp_workers_ready	Gauge	Ready subset of registered workers. Gap between this and workers_registered surfaces stale discovery or failed health checks

   Labels for all EPP metrics

   Const labels (bound once at startup, never from request body):

   • model — from set_served_model(), existing mechanism
   • inference_pool — from DYN_EPP_INFERENCE_POOL_NAME

   dynamo_epp_selection_errors_total adds reason (closed enum).
   dynamo_epp_workers_* adds worker_type (closed enum: aggregated/prefill/decode/encode).

   No worker_id, pod name, or endpoint labels on hot path metrics per the cardinality safety principle already in this DEP.
8. 0z5a commented on Sep 22, 2026

   

   0z5a

   on Sep 22, 2026

   More actions

   Hi @flpanbin @tmonty12, I'd like to take the DRT-free
   RouterRequestMetrics construction and standalone EPP integration
   slice: the proposed from_registry path, wiring into the existing
   private registry, and the corresponding compatibility, registry
   isolation, and observation-path tests.

   Following the proposed naming direction, this would reuse
   dynamo_component_router_* rather than introduce a parallel renamed
   set, while preserving the existing DRT/component path.

   The request-boundary instrumentation remains in #15056. The
   tokenizer/selection timing proposals also need to be reconciled
   with its existing phase instrumentation to avoid duplicate
   measurement paths.

   As a separate follow-up, I'd also like to take the
   workers_registered/workers_ready gauges, with the authoritative
   state source, counting unit, and readiness semantics agreed first.

   I'll avoid JSONL/config parity and picks_inflight, which
   @tryambak2019 has expressed interest in.

   Please flag any implementation already underway for the registry
   or worker-state slices so the scopes can be coordinated.

   👍React with 👍1Reacted by tk
9. flpanbin commented on Sep 22, 2026

   

   flpanbin

   on Sep 22, 2026

   Last edited by flpanbin

   ContributorAuthor

   More actions

   Hi @tmonty12 Thanks for the review. Point by point:

   1. from_registry — agreed. Reusing RouterRequestMetrics gives us the definitions, but it does not give us the observations. Most of those series is fed from inside RoutingHost / RequestGuard, none of which the EPP runs. So each series still needs an EPP-side observe call at a point where the EPP actually holds the data. That's why I listed "where (file::fn)" per row in the DEP. And even the definitions need one change. The help strings are hardcoded in from_component — six of the twelve say "observed at the router" or "processed by the router", which isn't true of the EPP, Frontend and router already share the name constant while writing their own help text, so could from_registry take the observer name, or a help override?

   2. Naming — would prefix = dynamo_epp be better than dynamo_component_router_?​
      Same struct, same buckets, only the prefix differs. Reasons:

      • Precedent inside Dynamo. Frontend and the KV router already observe the same request-level quantities from the same frontend_service::* constants, and emit two names, such as: dynamo_frontend_input_sequence_tokens vs dynamo_component_router_input_sequence_tokens. The EPP is a third observer, like the frontend, it has no dynamo_namespace / dynamo_component labels. So dynamo_epp_* follows the same rule.
      • dynamo_component_router_ implies a registered Dynamo component, which the EPP is not. It carries no dynamo_namespace / dynamo_component labels. Same name with a different label set silently breaks consumers, include the planner and Grafana, EPP metrics samples would be summed into the same distribution with no error.
      • llm-d does the same — its endpoint picker uses llm_d_epp_* for these quantities.

   3. On "ITL, KV hit rate, shared cache hit rate, non-max-overlap" come for free​ — free for registration, not for data. such as non-max-overlap / overlap_blocks_lost needs a crate change: the value is a scheduler-internal comparison (highest-overlap worker vs. the one selected) that isn't in the selection result. So may be we can register from_registry only a subset. Not every series in RouterRequestMetrics can be fed on day one, and their priorities differ. We can add the rest as each producer lands.

   4. Remaining metrics - still in progress I'm still working through the other domains (such as reservation lifecycle, Worker discovery and catalog, Disaggregated handoff). There are some details need to confirm, like the observation point, and which values a label can take. One example: For dynamo_epp_selection_errors_total you metioned, I noticed the SelectionError — bad_request / not_ready / not_found / conflict / internal / scheduler / sequence defined. The values in your comment don't match that set. But I think the reasons you suggested makes more sense. I have updated in DEP.
10. flpanbin

    changed the title [-][WIP] DEP: Standalone EPP observability [/-] [+]DEP: Standalone EPP observability [/+] on Sep 24, 2026
11. kvnloo commented on Oct 1, 2026

    

    kvnloo

    on Oct 1, 2026 – with ChatGPT Codex Connector

    More actions

    One implementation detail that may be worth making explicit in the DEP: lifecycle metrics should be cancellation-safe, not just response-path-safe.

    The current standalone EppRouter::pick() already has the right ownership shape for this: try_acquire_owned() gives a permit whose lifetime is exactly the admitted pick, including early error and future cancellation/drop. I would mirror that with an RAII metrics guard rather than pairing inc() / dec() at individual return sites.

    That gives clean semantics for picks_inflight:

    • semaphore rejection → increment an overload/shed counter, never inflight;
    • admitted pick → inflight +1;
    • success, selection error, tokenizer error, stale-worker rejection, or cancelled future → inflight -1 exactly once.

    I think the same principle should apply one level up at the ext_proc stream boundary. A request-duration / active-request guard should finalize on stream drop as well as normal response completion; otherwise client disconnects and Envoy cancellations disappear from the denominator or leak gauges.

    One other split I would keep: SelectionError reason describes the scheduler/selection failure; the ext_proc outcome describes the externally visible request result. They are related but shouldn't be forced into one label enum.

    This also keeps per-request identity in tracing/log fields rather than metric labels, consistent with the cardinality rules already in the DEP.

    AI assistance disclosure: I used an AI assistant to inspect the current EPP lifecycle/RAII paths and organize this note; I checked the referenced code paths before posting.
12. flpanbin commented on Oct 15, 2026

    

    flpanbin

    on Oct 15, 2026

    ContributorAuthor

    More actions

    One implementation detail that may be worth making explicit in the DEP: lifecycle metrics should be cancellation-safe, not just response-path-safe.

    The current standalone EppRouter::pick() already has the right ownership shape for this: try_acquire_owned() gives a permit whose lifetime is exactly the admitted pick, including early error and future cancellation/drop. I would mirror that with an RAII metrics guard rather than pairing inc() / dec() at individual return sites.

    That gives clean semantics for picks_inflight:

    • semaphore rejection → increment an overload/shed counter, never inflight;
    • admitted pick → inflight +1;
    • success, selection error, tokenizer error, stale-worker rejection, or cancelled future → inflight -1 exactly once.

    I think the same principle should apply one level up at the ext_proc stream boundary. A request-duration / active-request guard should finalize on stream drop as well as normal response completion; otherwise client disconnects and Envoy cancellations disappear from the denominator or leak gauges.

    One other split I would keep: SelectionError reason describes the scheduler/selection failure; the ext_proc outcome describes the externally visible request result. They are related but shouldn't be forced into one label enum.

    This also keeps per-request identity in tracing/log fields rather than metric labels, consistent with the cardinality rules already in the DEP.

    AI assistance disclosure: I used an AI assistant to inspect the current EPP lifecycle/RAII paths and organize this note; I checked the referenced code paths before posting.

    @kvnloo Thanks for the comments.

    lifecycle metrics should be cancellation-safe, not just response-path-safe.

    Agreed. dynamo_epp_requests_inflight now share the request-scope RAII guard with dynamo_epp_requests_total and dynamo_epp_request_duration_seconds.

    semaphore rejection → increment an overload/shed counter, never inflight;

    I added a extra dynamo_epp_request_rejection_total on PickError::Overloaded; shed requests never touch picks_inflight.

    ❤️React with ❤️1Reacted by Kevin Rajan