diff --git a/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md b/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md index 70081fb3cb8..55633996898 100644 --- a/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md +++ b/content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md @@ -150,55 +150,40 @@ These metrics are available for sources, processors, and destinations. - Use the `component_type` tag to filter or group by the type of source, processor, or destination, such as `quota` for the Quota processor. - Use the `component_kind` tag to filter or group by `source`, `transform` (processor) or `sink` (destination). +{{< tabs >}} +{{% tab "Sources" %}} + +### Throughput + +Bytes in +: **Metric**: `pipelines.component_received_bytes_total` +: **Description**: The number of raw bytes read from the source's input, before any decoding or transformation. + Events in : **Metric**: `pipelines.component_received_events_total` : **Description**: The number of events received by the component. -: **Available for**: Sources, processors, and destinations. Events out : **Metric**: `pipelines.component_sent_events_total` : **Description**: The number of events the component sends downstream. -: **Available for**: Sources, processors, and destinations. Event bytes in : **Metric**: `pipelines.component_received_event_bytes_total` : **Description**: The byte size of events received by the component. -: **Available for**: Sources, processors, and destinations. Event bytes out : **Metric**: `pipelines.component_sent_event_bytes_total` : **Description**: The byte size of events the component sends downstream. -: **Available for**: Sources, processors, and destinations. -Bytes in -: **Metric**: `pipelines.component_received_bytes_total` -: **Description**: The number of raw bytes read from the source's input, before any decoding or transformation. -: **Available for**: Sources. - -Bytes out -: **Metric**: `pipelines.component_sent_bytes_total` -: **Description**: The number of raw bytes written to the destination's output, after encoding and transformations. -: **Available for**: Destinations. - -Events included -: **Metric**: `pipelines.included_events_total` -: **Description**: The number of events that matched the processor's filter query and were processed. Events that do not match the filter query skip the processor and continue to the next component. -: **Available for**: Processors. - -Event bytes included -: **Metric**: `pipelines.included_event_bytes_total` -: **Description**: The byte size of events that matched the processor's filter query and were processed. -: **Available for**: Processors. +### Errors, data dropped, and timed outs Errors : **Metric**: `pipelines.component_errors_total` : **Description**: The number of errors encountered by the component. Depending on the component, this metric can include an `error_code`, `error_type`, or `reason` tag that describes the error. -: **Available for**: Sources, processors, and destinations. Data dropped intentionally or unintentionally : **Metric**: `pipelines.component_discarded_events_total` : **Description**: The number of events dropped. **Note**: To break down this metric, use the `intentional:true` tag to filter for events that are intentionally dropped or the `intentional:false` tag for events that are not intentionally dropped. -: **Available for**: Sources, processors, and destinations. Timed out events : **Metric**: `pipelines.component_timed_out_events_total` @@ -210,15 +195,66 @@ Timed out requests : **Description**: The number of requests that timed out for sources that send events to the Worker in batches using HTTP requests. : **Available for**: HTTP-based sources that have a configured timeout, such as the Datadog Agent. +### Performance + +Send latency +: **Metric**: `pipelines.source_send_latency_seconds` +: **Description**: The time it takes for the source to send a chunk of events to the next component. Available in Worker version 2.16 and later. + +Send batch latency +: **Metric**: `pipelines.source_send_batch_latency_seconds` +: **Description**: The time it takes for the source to send a batch, which can contain multiple event chunks, to the next component. Available in Worker version 2.16 and later. + Source lag time : **Metric**: `pipelines.source_lag_time_seconds` : **Description**: The difference, in seconds, between an event's own timestamp and when the Worker received it. High values indicate stale or delayed data arriving at the pipeline. -: **Available for**: Sources. -Utilization -: **Metric**: `pipelines.utilization` -: **Description**: The component's activity. A value of `0` indicates an idle component that is waiting for input. A value close to `1` indicates a component that is never idle, meaning that the component is likely a bottleneck in the processing topology that is creating backpressure. This might cause events to be dropped. -: **Available for**: Processors and destinations. +### Buffer + +Use these metrics to analyze buffer performance. All metrics are emitted on a one-second interval, unless otherwise stated. + +{{% observability_pipelines/metrics/buffer/sources %}} + +{{% /tab %}} +{{% tab "Processors" %}} + +### Throughput + +Events in +: **Metric**: `pipelines.component_received_events_total` +: **Description**: The number of events received by the component. + +Events out +: **Metric**: `pipelines.component_sent_events_total` +: **Description**: The number of events the component sends downstream. + +Event bytes in +: **Metric**: `pipelines.component_received_event_bytes_total` +: **Description**: The byte size of events received by the component. + +Event bytes out +: **Metric**: `pipelines.component_sent_event_bytes_total` +: **Description**: The byte size of events the component sends downstream. + +Events included +: **Metric**: `pipelines.included_events_total` +: **Description**: The number of events that matched the processor's filter query and were processed. Events that do not match the filter query skip the processor and continue to the next component. + +Event bytes included +: **Metric**: `pipelines.included_event_bytes_total` +: **Description**: The byte size of events that matched the processor's filter query and were processed. + +### Errors and data dropped + +Errors +: **Metric**: `pipelines.component_errors_total` +: **Description**: The number of errors encountered by the component. Depending on the component, this metric can include an `error_code`, `error_type`, or `reason` tag that describes the error. + +Data dropped intentionally or unintentionally +: **Metric**: `pipelines.component_discarded_events_total` +: **Description**: The number of events dropped. **Note**: To break down this metric, use the `intentional:true` tag to filter for events that are intentionally dropped or the `intentional:false` tag for events that are not intentionally dropped. + +### Performance CPU usage : **Metric**: `pipelines.component_cpu_usage_ns_total` @@ -226,15 +262,69 @@ CPU usage : **Available for these log processors**:
- Custom Processor
- Dedupe
- Enrichment Table
- Grok Parser
- Parse JSON
- Parse XML
- Reduce
- Remap to OCSF
- Sensitive Data Scanner
- Split Array
- Throttle log processors : **Available for these metrics processors**:
- Aggregate
- Tag Cardinality Limit metrics -Send latency -: **Metric**: `pipelines.source_send_latency_seconds` -: **Description**: The time it takes for the source to send a chunk of events to the next component. Available in Worker version 2.16 and later. -: **Available for**: Sources. +Utilization +: **Metric**: `pipelines.utilization` +: **Description**: The component's activity. A value of `0` indicates an idle component that is waiting for input. A value close to `1` indicates a component that is never idle, meaning that the component is likely a bottleneck in the processing topology that is creating backpressure. This might cause events to be dropped. -Send batch latency -: **Metric**: `pipelines.source_send_batch_latency_seconds` -: **Description**: The time it takes for the source to send a batch, which can contain multiple event chunks, to the next component. Available in Worker version 2.16 and later. -: **Available for**: Sources. +### Buffer + +Use these metrics to analyze buffer performance. All metrics are emitted on a one-second interval, unless otherwise stated. + +{{% observability_pipelines/metrics/buffer/processors %}} + +{{% /tab %}} +{{% tab "Destinations" %}} + +### Throughput + +Bytes out +: **Metric**: `pipelines.component_sent_bytes_total` +: **Description**: The number of raw bytes written to the destination's output, after encoding and transformations. + +Events in +: **Metric**: `pipelines.component_received_events_total` +: **Description**: The number of events received by the component. + +Events out +: **Metric**: `pipelines.component_sent_events_total` +: **Description**: The number of events the component sends downstream. + +Event bytes in +: **Metric**: `pipelines.component_received_event_bytes_total` +: **Description**: The byte size of events received by the component. + +Event bytes out +: **Metric**: `pipelines.component_sent_event_bytes_total` +: **Description**: The byte size of events the component sends downstream. + +### Errors and data dropped + +Errors +: **Metric**: `pipelines.component_errors_total` +: **Description**: The number of errors encountered by the component. Depending on the component, this metric can include an `error_code`, `error_type`, or `reason` tag that describes the error. + +Data dropped intentionally or unintentionally +: **Metric**: `pipelines.component_discarded_events_total` +: **Description**: The number of events dropped. **Note**: To break down this metric, use the `intentional:true` tag to filter for events that are intentionally dropped or the `intentional:false` tag for events that are not intentionally dropped. + +### Performance + +Utilization +: **Metric**: `pipelines.utilization` +: **Description**: The component's activity. A value of `0` indicates an idle component that is waiting for input. A value close to `1` indicates a component that is never idle, meaning that the component is likely a bottleneck in the processing topology that is creating backpressure. This might cause events to be dropped. + +### Buffer + +Use these metrics to analyze buffer performance. All metrics are emitted on a one-second interval, unless otherwise stated. + +{{% observability_pipelines/metrics/buffer/destinations %}} + +#### Deprecated buffer metrics + +{{% observability_pipelines/metrics/buffer/deprecated_destination_metrics %}} + +{{% /tab %}} +{{< /tabs >}} ## HTTP server metrics @@ -339,26 +429,6 @@ These metrics provide information about the adaptive concurrency controller, whi : **Description**: The historical mean RTT, in seconds, for HTTP requests to this destination, used as the long-term baseline for adaptive concurrency adjustments. : **Metric type**: distribution -## Buffer metrics - -Use these metrics to analyze buffer performance. All metrics are emitted on a one-second interval, unless otherwise stated. - -### Source buffer metrics - -{{% observability_pipelines/metrics/buffer/sources %}} - -### Processor buffer metrics - -{{% observability_pipelines/metrics/buffer/processors %}} - -### Destination buffer metrics - -{{% observability_pipelines/metrics/buffer/destinations %}} - -### Deprecated buffer metrics - -{{% observability_pipelines/metrics/buffer/deprecated_destination_metrics %}} - ## Further reading {{< partial name="whats-next/whats-next.html" >}}