From 0d59c92c91f83db027c17b4f5aa638abd0282373 Mon Sep 17 00:00:00 2001 From: May Lee Date: Fri, 17 Jul 2026 15:15:47 -0400 Subject: [PATCH 1/2] add tabs --- .../pipeline_usage_metrics.md | 152 ++++++++++++------ 1 file changed, 101 insertions(+), 51 deletions(-) diff --git a/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md b/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md index db6c829fda9..d0fc6fce18b 100644 --- a/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md +++ b/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md @@ -150,91 +150,141 @@ These metrics are available for sources, processors, and destinations. - Use the `component_type` tag to filter or group by the type of source, processor, or destination, such as `quota` for the Quota processor. - Use the `component_kind` tag to filter or group by `source`, `transform` (processor) or `sink` (destination). +{{< tabs >}} +{{% tab "Sources" %}} + +Bytes in +: **Metric**: `pipelines.component_received_bytes_total` +: **Description**: The number of raw bytes read from the source's input, before any decoding or transformation. + +Data dropped intentionally or unintentionally +: **Metric**: `pipelines.component_discarded_events_total` +: **Description**: The number of events dropped. **Note**: To break down this metric, use the `intentional:true` tag to filter for events that are intentionally dropped or the `intentional:false` tag for events that are not intentionally dropped. + +Errors +: **Metric**: `pipelines.component_errors_total` +: **Description**: The number of errors encountered by the component. Depending on the component, this metric can include an `error_code`, `error_type`, or `reason` tag that describes the error. + +Event bytes in +: **Metric**: `pipelines.component_received_event_bytes_total` +: **Description**: The byte size of events received by the component. + +Event bytes out +: **Metric**: `pipelines.component_sent_event_bytes_total` +: **Description**: The byte size of events the component sends downstream. + Events in : **Metric**: `pipelines.component_received_events_total` : **Description**: The number of events received by the component. -: **Available for**: Sources, processors, and destinations. Events out : **Metric**: `pipelines.component_sent_events_total` : **Description**: The number of events the component sends downstream. -: **Available for**: Sources, processors, and destinations. + +Send batch latency +: **Metric**: `pipelines.source_send_batch_latency_seconds` +: **Description**: The time it takes for the source to send a batch, which can contain multiple event chunks, to the next component. Available in Worker version 2.16 and later. + +Send latency +: **Metric**: `pipelines.source_send_latency_seconds` +: **Description**: The time it takes for the source to send a chunk of events to the next component. Available in Worker version 2.16 and later. + +Source lag time +: **Metric**: `pipelines.source_lag_time_seconds` +: **Description**: The difference, in seconds, between an event's own timestamp and when the Worker received it. High values indicate stale or delayed data arriving at the pipeline. + +Timed out events +: **Metric**: `pipelines.component_timed_out_events_total` +: **Description**: The number of events that waited more than 5 seconds to be sent to the first processor and resulted in a HTTP 503 error. This could happen when delivery of events are blocked. +: **Available for**: HTTP-based sources that have a configured timeout, such as the Datadog Agent. + +Timed out requests +: **Metric**: `pipelines.component_timed_out_requests_total` +: **Description**: The number of requests that timed out for sources that send events to the Worker in batches using HTTP requests. +: **Available for**: HTTP-based sources that have a configured timeout, such as the Datadog Agent. + +{{% /tab %}} +{{% tab "Processors" %}} + +CPU usage +: **Metric**: `pipelines.component_cpu_usage_ns_total` +: **Description**: The CPU time consumed by a component, in nanoseconds. Use this metric to attribute CPU cost to individual processors. Available in Worker version 2.18 and later for Linux and MacOS. +: **Available for these log processors**:
- Custom Processor
- Dedupe
- Enrichment Table
- Grok Parser
- Parse JSON
- Parse XML
- Reduce
- Remap to OCSF
- Sensitive Data Scanner
- Split Array
- Throttle log processors +: **Available for these metrics processors**:
- Aggregate
- Tag Cardinality Limit metrics + +Data dropped intentionally or unintentionally +: **Metric**: `pipelines.component_discarded_events_total` +: **Description**: The number of events dropped. **Note**: To break down this metric, use the `intentional:true` tag to filter for events that are intentionally dropped or the `intentional:false` tag for events that are not intentionally dropped. + +Errors +: **Metric**: `pipelines.component_errors_total` +: **Description**: The number of errors encountered by the component. Depending on the component, this metric can include an `error_code`, `error_type`, or `reason` tag that describes the error. Event bytes in : **Metric**: `pipelines.component_received_event_bytes_total` : **Description**: The byte size of events received by the component. -: **Available for**: Sources, processors, and destinations. + +Event bytes included +: **Metric**: `pipelines.included_event_bytes_total` +: **Description**: The byte size of events that matched the processor's filter query and were processed. Event bytes out : **Metric**: `pipelines.component_sent_event_bytes_total` : **Description**: The byte size of events the component sends downstream. -: **Available for**: Sources, processors, and destinations. - -Bytes in -: **Metric**: `pipelines.component_received_bytes_total` -: **Description**: The number of raw bytes read from the source's input, before any decoding or transformation. -: **Available for**: Sources. -Bytes out -: **Metric**: `pipelines.component_sent_bytes_total` -: **Description**: The number of raw bytes written to the destination's output, after encoding and transformations. -: **Available for**: Destinations. +Events in +: **Metric**: `pipelines.component_received_events_total` +: **Description**: The number of events received by the component. Events included : **Metric**: `pipelines.included_events_total` : **Description**: The number of events that matched the processor's filter query and were processed. Events that do not match the filter query skip the processor and continue to the next component. -: **Available for**: Processors. -Event bytes included -: **Metric**: `pipelines.included_event_bytes_total` -: **Description**: The byte size of events that matched the processor's filter query and were processed. -: **Available for**: Processors. +Events out +: **Metric**: `pipelines.component_sent_events_total` +: **Description**: The number of events the component sends downstream. -Errors -: **Metric**: `pipelines.component_errors_total` -: **Description**: The number of errors encountered by the component. Depending on the component, this metric can include an `error_code`, `error_type`, or `reason` tag that describes the error. -: **Available for**: Sources, processors, and destinations. +Utilization +: **Metric**: `pipelines.utilization` +: **Description**: The component's activity. A value of `0` indicates an idle component that is waiting for input. A value close to `1` indicates a component that is never idle, meaning that the component is likely a bottleneck in the processing topology that is creating backpressure. This might cause events to be dropped. + +{{% /tab %}} +{{% tab "Destinations" %}} + +Bytes out +: **Metric**: `pipelines.component_sent_bytes_total` +: **Description**: The number of raw bytes written to the destination's output, after encoding and transformations. Data dropped intentionally or unintentionally : **Metric**: `pipelines.component_discarded_events_total` : **Description**: The number of events dropped. **Note**: To break down this metric, use the `intentional:true` tag to filter for events that are intentionally dropped or the `intentional:false` tag for events that are not intentionally dropped. -: **Available for**: Sources, processors, and destinations. -Timed out events -: **Metric**: `pipelines.component_timed_out_events_total` -: **Description**: The number of events that waited more than 5 seconds to be sent to the first processor and resulted in a HTTP 503 error. This could happen when delivery of events are blocked. -: **Available for**: HTTP-based sources that have a configured timeout, such as the Datadog Agent. +Errors +: **Metric**: `pipelines.component_errors_total` +: **Description**: The number of errors encountered by the component. Depending on the component, this metric can include an `error_code`, `error_type`, or `reason` tag that describes the error. -Timed out requests -: **Metric**: `pipelines.component_timed_out_requests_total` -: **Description**: The number of requests that timed out for sources that send events to the Worker in batches using HTTP requests. -: **Available for**: HTTP-based sources that have a configured timeout, such as the Datadog Agent. +Event bytes in +: **Metric**: `pipelines.component_received_event_bytes_total` +: **Description**: The byte size of events received by the component. -Source lag time -: **Metric**: `pipelines.source_lag_time_seconds` -: **Description**: The difference, in seconds, between an event's own timestamp and when the Worker received it. High values indicate stale or delayed data arriving at the pipeline. -: **Available for**: Sources. +Event bytes out +: **Metric**: `pipelines.component_sent_event_bytes_total` +: **Description**: The byte size of events the component sends downstream. + +Events in +: **Metric**: `pipelines.component_received_events_total` +: **Description**: The number of events received by the component. + +Events out +: **Metric**: `pipelines.component_sent_events_total` +: **Description**: The number of events the component sends downstream. Utilization : **Metric**: `pipelines.utilization` : **Description**: The component's activity. A value of `0` indicates an idle component that is waiting for input. A value close to `1` indicates a component that is never idle, meaning that the component is likely a bottleneck in the processing topology that is creating backpressure. This might cause events to be dropped. -: **Available for**: Processors and destinations. - -CPU usage -: **Metric**: `pipelines.component_cpu_usage_ns_total` -: **Description**: The CPU time consumed by a component, in nanoseconds. Use this metric to attribute CPU cost to individual processors. Available in Worker version 2.18 and later for Linux and MacOS. -: **Available for these log processors**:
- Custom Processor
- Dedupe
- Enrichment Table
- Grok Parser
- Parse JSON
- Parse XML
- Reduce
- Remap to OCSF
- Sensitive Data Scanner
- Split Array
- Throttle log processors -: **Available for these metrics processors**:
- Aggregate
- Tag Cardinality Limit metrics - -Send latency -: **Metric**: `pipelines.source_send_latency_seconds` -: **Description**: The time it takes for the source to send a chunk of events to the next component. Available in Worker version 2.16 and later. -: **Available for**: Sources. -Send batch latency -: **Metric**: `pipelines.source_send_batch_latency_seconds` -: **Description**: The time it takes for the source to send a batch, which can contain multiple event chunks, to the next component. Available in Worker version 2.16 and later. -: **Available for**: Sources. +{{% /tab %}} +{{< /tabs >}} ## HTTP server metrics From e5a1134c21332605bc67a414237bae7f3c5054f4 Mon Sep 17 00:00:00 2001 From: May Lee Date: Mon, 20 Jul 2026 15:26:51 -0400 Subject: [PATCH 2/2] add headers and move buffers --- .../pipeline_usage_metrics.md | 174 ++++++++++-------- 1 file changed, 97 insertions(+), 77 deletions(-) diff --git a/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md b/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md index 4535257ec89..55633996898 100644 --- a/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md +++ b/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md @@ -153,17 +153,19 @@ These metrics are available for sources, processors, and destinations. {{< tabs >}} {{% tab "Sources" %}} +### Throughput + Bytes in : **Metric**: `pipelines.component_received_bytes_total` : **Description**: The number of raw bytes read from the source's input, before any decoding or transformation. -Data dropped intentionally or unintentionally -: **Metric**: `pipelines.component_discarded_events_total` -: **Description**: The number of events dropped. **Note**: To break down this metric, use the `intentional:true` tag to filter for events that are intentionally dropped or the `intentional:false` tag for events that are not intentionally dropped. +Events in +: **Metric**: `pipelines.component_received_events_total` +: **Description**: The number of events received by the component. -Errors -: **Metric**: `pipelines.component_errors_total` -: **Description**: The number of errors encountered by the component. Depending on the component, this metric can include an `error_code`, `error_type`, or `reason` tag that describes the error. +Events out +: **Metric**: `pipelines.component_sent_events_total` +: **Description**: The number of events the component sends downstream. Event bytes in : **Metric**: `pipelines.component_received_event_bytes_total` @@ -173,25 +175,15 @@ Event bytes out : **Metric**: `pipelines.component_sent_event_bytes_total` : **Description**: The byte size of events the component sends downstream. -Events in -: **Metric**: `pipelines.component_received_events_total` -: **Description**: The number of events received by the component. - -Events out -: **Metric**: `pipelines.component_sent_events_total` -: **Description**: The number of events the component sends downstream. +### Errors, data dropped, and timed outs -Send batch latency -: **Metric**: `pipelines.source_send_batch_latency_seconds` -: **Description**: The time it takes for the source to send a batch, which can contain multiple event chunks, to the next component. Available in Worker version 2.16 and later. - -Send latency -: **Metric**: `pipelines.source_send_latency_seconds` -: **Description**: The time it takes for the source to send a chunk of events to the next component. Available in Worker version 2.16 and later. +Errors +: **Metric**: `pipelines.component_errors_total` +: **Description**: The number of errors encountered by the component. Depending on the component, this metric can include an `error_code`, `error_type`, or `reason` tag that describes the error. -Source lag time -: **Metric**: `pipelines.source_lag_time_seconds` -: **Description**: The difference, in seconds, between an event's own timestamp and when the Worker received it. High values indicate stale or delayed data arriving at the pipeline. +Data dropped intentionally or unintentionally +: **Metric**: `pipelines.component_discarded_events_total` +: **Description**: The number of events dropped. **Note**: To break down this metric, use the `intentional:true` tag to filter for events that are intentionally dropped or the `intentional:false` tag for events that are not intentionally dropped. Timed out events : **Metric**: `pipelines.component_timed_out_events_total` @@ -203,65 +195,99 @@ Timed out requests : **Description**: The number of requests that timed out for sources that send events to the Worker in batches using HTTP requests. : **Available for**: HTTP-based sources that have a configured timeout, such as the Datadog Agent. +### Performance + +Send latency +: **Metric**: `pipelines.source_send_latency_seconds` +: **Description**: The time it takes for the source to send a chunk of events to the next component. Available in Worker version 2.16 and later. + +Send batch latency +: **Metric**: `pipelines.source_send_batch_latency_seconds` +: **Description**: The time it takes for the source to send a batch, which can contain multiple event chunks, to the next component. Available in Worker version 2.16 and later. + +Source lag time +: **Metric**: `pipelines.source_lag_time_seconds` +: **Description**: The difference, in seconds, between an event's own timestamp and when the Worker received it. High values indicate stale or delayed data arriving at the pipeline. + +### Buffer + +Use these metrics to analyze buffer performance. All metrics are emitted on a one-second interval, unless otherwise stated. + +{{% observability_pipelines/metrics/buffer/sources %}} + {{% /tab %}} {{% tab "Processors" %}} -CPU usage -: **Metric**: `pipelines.component_cpu_usage_ns_total` -: **Description**: The CPU time consumed by a component, in nanoseconds. Use this metric to attribute CPU cost to individual processors. Available in Worker version 2.18 and later for Linux and MacOS. -: **Available for these log processors**:
- Custom Processor
- Dedupe
- Enrichment Table
- Grok Parser
- Parse JSON
- Parse XML
- Reduce
- Remap to OCSF
- Sensitive Data Scanner
- Split Array
- Throttle log processors -: **Available for these metrics processors**:
- Aggregate
- Tag Cardinality Limit metrics +### Throughput -Data dropped intentionally or unintentionally -: **Metric**: `pipelines.component_discarded_events_total` -: **Description**: The number of events dropped. **Note**: To break down this metric, use the `intentional:true` tag to filter for events that are intentionally dropped or the `intentional:false` tag for events that are not intentionally dropped. +Events in +: **Metric**: `pipelines.component_received_events_total` +: **Description**: The number of events received by the component. -Errors -: **Metric**: `pipelines.component_errors_total` -: **Description**: The number of errors encountered by the component. Depending on the component, this metric can include an `error_code`, `error_type`, or `reason` tag that describes the error. +Events out +: **Metric**: `pipelines.component_sent_events_total` +: **Description**: The number of events the component sends downstream. Event bytes in : **Metric**: `pipelines.component_received_event_bytes_total` : **Description**: The byte size of events received by the component. -Event bytes included -: **Metric**: `pipelines.included_event_bytes_total` -: **Description**: The byte size of events that matched the processor's filter query and were processed. - Event bytes out : **Metric**: `pipelines.component_sent_event_bytes_total` : **Description**: The byte size of events the component sends downstream. -Events in -: **Metric**: `pipelines.component_received_events_total` -: **Description**: The number of events received by the component. - Events included : **Metric**: `pipelines.included_events_total` : **Description**: The number of events that matched the processor's filter query and were processed. Events that do not match the filter query skip the processor and continue to the next component. -Events out -: **Metric**: `pipelines.component_sent_events_total` -: **Description**: The number of events the component sends downstream. +Event bytes included +: **Metric**: `pipelines.included_event_bytes_total` +: **Description**: The byte size of events that matched the processor's filter query and were processed. + +### Errors and data dropped + +Errors +: **Metric**: `pipelines.component_errors_total` +: **Description**: The number of errors encountered by the component. Depending on the component, this metric can include an `error_code`, `error_type`, or `reason` tag that describes the error. + +Data dropped intentionally or unintentionally +: **Metric**: `pipelines.component_discarded_events_total` +: **Description**: The number of events dropped. **Note**: To break down this metric, use the `intentional:true` tag to filter for events that are intentionally dropped or the `intentional:false` tag for events that are not intentionally dropped. + +### Performance + +CPU usage +: **Metric**: `pipelines.component_cpu_usage_ns_total` +: **Description**: The CPU time consumed by a component, in nanoseconds. Use this metric to attribute CPU cost to individual processors. Available in Worker version 2.18 and later for Linux and MacOS. +: **Available for these log processors**:
- Custom Processor
- Dedupe
- Enrichment Table
- Grok Parser
- Parse JSON
- Parse XML
- Reduce
- Remap to OCSF
- Sensitive Data Scanner
- Split Array
- Throttle log processors +: **Available for these metrics processors**:
- Aggregate
- Tag Cardinality Limit metrics Utilization : **Metric**: `pipelines.utilization` : **Description**: The component's activity. A value of `0` indicates an idle component that is waiting for input. A value close to `1` indicates a component that is never idle, meaning that the component is likely a bottleneck in the processing topology that is creating backpressure. This might cause events to be dropped. +### Buffer + +Use these metrics to analyze buffer performance. All metrics are emitted on a one-second interval, unless otherwise stated. + +{{% observability_pipelines/metrics/buffer/processors %}} + {{% /tab %}} {{% tab "Destinations" %}} +### Throughput + Bytes out : **Metric**: `pipelines.component_sent_bytes_total` : **Description**: The number of raw bytes written to the destination's output, after encoding and transformations. -Data dropped intentionally or unintentionally -: **Metric**: `pipelines.component_discarded_events_total` -: **Description**: The number of events dropped. **Note**: To break down this metric, use the `intentional:true` tag to filter for events that are intentionally dropped or the `intentional:false` tag for events that are not intentionally dropped. +Events in +: **Metric**: `pipelines.component_received_events_total` +: **Description**: The number of events received by the component. -Errors -: **Metric**: `pipelines.component_errors_total` -: **Description**: The number of errors encountered by the component. Depending on the component, this metric can include an `error_code`, `error_type`, or `reason` tag that describes the error. +Events out +: **Metric**: `pipelines.component_sent_events_total` +: **Description**: The number of events the component sends downstream. Event bytes in : **Metric**: `pipelines.component_received_event_bytes_total` @@ -271,18 +297,32 @@ Event bytes out : **Metric**: `pipelines.component_sent_event_bytes_total` : **Description**: The byte size of events the component sends downstream. -Events in -: **Metric**: `pipelines.component_received_events_total` -: **Description**: The number of events received by the component. +### Errors and data dropped -Events out -: **Metric**: `pipelines.component_sent_events_total` -: **Description**: The number of events the component sends downstream. +Errors +: **Metric**: `pipelines.component_errors_total` +: **Description**: The number of errors encountered by the component. Depending on the component, this metric can include an `error_code`, `error_type`, or `reason` tag that describes the error. + +Data dropped intentionally or unintentionally +: **Metric**: `pipelines.component_discarded_events_total` +: **Description**: The number of events dropped. **Note**: To break down this metric, use the `intentional:true` tag to filter for events that are intentionally dropped or the `intentional:false` tag for events that are not intentionally dropped. + +### Performance Utilization : **Metric**: `pipelines.utilization` : **Description**: The component's activity. A value of `0` indicates an idle component that is waiting for input. A value close to `1` indicates a component that is never idle, meaning that the component is likely a bottleneck in the processing topology that is creating backpressure. This might cause events to be dropped. +### Buffer + +Use these metrics to analyze buffer performance. All metrics are emitted on a one-second interval, unless otherwise stated. + +{{% observability_pipelines/metrics/buffer/destinations %}} + +#### Deprecated buffer metrics + +{{% observability_pipelines/metrics/buffer/deprecated_destination_metrics %}} + {{% /tab %}} {{< /tabs >}} @@ -389,26 +429,6 @@ These metrics provide information about the adaptive concurrency controller, whi : **Description**: The historical mean RTT, in seconds, for HTTP requests to this destination, used as the long-term baseline for adaptive concurrency adjustments. : **Metric type**: distribution -## Buffer metrics - -Use these metrics to analyze buffer performance. All metrics are emitted on a one-second interval, unless otherwise stated. - -### Source buffer metrics - -{{% observability_pipelines/metrics/buffer/sources %}} - -### Processor buffer metrics - -{{% observability_pipelines/metrics/buffer/processors %}} - -### Destination buffer metrics - -{{% observability_pipelines/metrics/buffer/destinations %}} - -### Deprecated buffer metrics - -{{% observability_pipelines/metrics/buffer/deprecated_destination_metrics %}} - ## Further reading {{< partial name="whats-next/whats-next.html" >}}