Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -150,55 +150,40 @@
- Use the `component_type` tag to filter or group by the type of source, processor, or destination, such as `quota` for the Quota processor.
- Use the `component_kind` tag to filter or group by `source`, `transform` (processor) or `sink` (destination).

{{< tabs >}}
{{% tab "Sources" %}}

### Throughput

Bytes in
: **Metric**: `pipelines.component_received_bytes_total`
: **Description**: The number of raw bytes read from the source's input, before any decoding or transformation.

Events in
: **Metric**: `pipelines.component_received_events_total`
: **Description**: The number of events received by the component.
: **Available for**: Sources, processors, and destinations.

Events out
: **Metric**: `pipelines.component_sent_events_total`
: **Description**: The number of events the component sends downstream.
: **Available for**: Sources, processors, and destinations.

Event bytes in
: **Metric**: `pipelines.component_received_event_bytes_total`
: **Description**: The byte size of events received by the component.
: **Available for**: Sources, processors, and destinations.

Event bytes out
: **Metric**: `pipelines.component_sent_event_bytes_total`
: **Description**: The byte size of events the component sends downstream.
: **Available for**: Sources, processors, and destinations.

Bytes in
: **Metric**: `pipelines.component_received_bytes_total`
: **Description**: The number of raw bytes read from the source's input, before any decoding or transformation.
: **Available for**: Sources.

Bytes out
: **Metric**: `pipelines.component_sent_bytes_total`
: **Description**: The number of raw bytes written to the destination's output, after encoding and transformations.
: **Available for**: Destinations.

Events included
: **Metric**: `pipelines.included_events_total`
: **Description**: The number of events that matched the processor's filter query and were processed. Events that do not match the filter query skip the processor and continue to the next component.
: **Available for**: Processors.

Event bytes included
: **Metric**: `pipelines.included_event_bytes_total`
: **Description**: The byte size of events that matched the processor's filter query and were processed.
: **Available for**: Processors.
### Errors, data dropped, and timed outs

Errors
: **Metric**: `pipelines.component_errors_total`
: **Description**: The number of errors encountered by the component. Depending on the component, this metric can include an `error_code`, `error_type`, or `reason` tag that describes the error.
: **Available for**: Sources, processors, and destinations.

Data dropped intentionally or unintentionally
: **Metric**: `pipelines.component_discarded_events_total`
: **Description**: The number of events dropped. **Note**: To break down this metric, use the `intentional:true` tag to filter for events that are intentionally dropped or the `intentional:false` tag for events that are not intentionally dropped.
: **Available for**: Sources, processors, and destinations.

Timed out events
: **Metric**: `pipelines.component_timed_out_events_total`
Expand All @@ -210,31 +195,136 @@
: **Description**: The number of requests that timed out for sources that send events to the Worker in batches using HTTP requests.
: **Available for**: HTTP-based sources that have a configured timeout, such as the Datadog Agent.

### Performance

Send latency
: **Metric**: `pipelines.source_send_latency_seconds`
: **Description**: The time it takes for the source to send a chunk of events to the next component. Available in Worker version 2.16 and later.

Send batch latency

Check notice on line 204 in content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.sentencelength

Suggestion: Try to keep your sentence length to 25 words or fewer.
: **Metric**: `pipelines.source_send_batch_latency_seconds`
: **Description**: The time it takes for the source to send a batch, which can contain multiple event chunks, to the next component. Available in Worker version 2.16 and later.

Source lag time
: **Metric**: `pipelines.source_lag_time_seconds`
: **Description**: The difference, in seconds, between an event's own timestamp and when the Worker received it. High values indicate stale or delayed data arriving at the pipeline.
: **Available for**: Sources.

Utilization
: **Metric**: `pipelines.utilization`
: **Description**: The component's activity. A value of `0` indicates an idle component that is waiting for input. A value close to `1` indicates a component that is never idle, meaning that the component is likely a bottleneck in the processing topology that is creating backpressure. This might cause events to be dropped.
: **Available for**: Processors and destinations.
### Buffer

Use these metrics to analyze buffer performance. All metrics are emitted on a one-second interval, unless otherwise stated.

{{% observability_pipelines/metrics/buffer/sources %}}

{{% /tab %}}
{{% tab "Processors" %}}

### Throughput

Events in
: **Metric**: `pipelines.component_received_events_total`
: **Description**: The number of events received by the component.

Events out
: **Metric**: `pipelines.component_sent_events_total`
: **Description**: The number of events the component sends downstream.

Event bytes in
: **Metric**: `pipelines.component_received_event_bytes_total`
: **Description**: The byte size of events received by the component.

Event bytes out
: **Metric**: `pipelines.component_sent_event_bytes_total`
: **Description**: The byte size of events the component sends downstream.

Events included
: **Metric**: `pipelines.included_events_total`
: **Description**: The number of events that matched the processor's filter query and were processed. Events that do not match the filter query skip the processor and continue to the next component.

Event bytes included
: **Metric**: `pipelines.included_event_bytes_total`
: **Description**: The byte size of events that matched the processor's filter query and were processed.

### Errors and data dropped

Errors
: **Metric**: `pipelines.component_errors_total`
: **Description**: The number of errors encountered by the component. Depending on the component, this metric can include an `error_code`, `error_type`, or `reason` tag that describes the error.

Data dropped intentionally or unintentionally
: **Metric**: `pipelines.component_discarded_events_total`
: **Description**: The number of events dropped. **Note**: To break down this metric, use the `intentional:true` tag to filter for events that are intentionally dropped or the `intentional:false` tag for events that are not intentionally dropped.

Check notice on line 255 in content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.sentencelength

Suggestion: Try to keep your sentence length to 25 words or fewer.

### Performance

CPU usage
: **Metric**: `pipelines.component_cpu_usage_ns_total`
: **Description**: The CPU time consumed by a component, in nanoseconds. Use this metric to attribute CPU cost to individual processors. Available in Worker version 2.18 and later for Linux and MacOS.
: **Available for these log processors**:<br>- Custom Processor<br>- Dedupe<br>- Enrichment Table<br>- Grok Parser<br>- Parse JSON<br>- Parse XML<br>- Reduce<br>- Remap to OCSF<br>- Sensitive Data Scanner<br>- Split Array<br>- Throttle log processors
: **Available for these metrics processors**:<br>- Aggregate <br>- Tag Cardinality Limit metrics

Send latency
: **Metric**: `pipelines.source_send_latency_seconds`
: **Description**: The time it takes for the source to send a chunk of events to the next component. Available in Worker version 2.16 and later.
: **Available for**: Sources.
Utilization
: **Metric**: `pipelines.utilization`
: **Description**: The component's activity. A value of `0` indicates an idle component that is waiting for input. A value close to `1` indicates a component that is never idle, meaning that the component is likely a bottleneck in the processing topology that is creating backpressure. This might cause events to be dropped.

Check notice on line 267 in content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.sentencelength

Suggestion: Try to keep your sentence length to 25 words or fewer.

Send batch latency
: **Metric**: `pipelines.source_send_batch_latency_seconds`
: **Description**: The time it takes for the source to send a batch, which can contain multiple event chunks, to the next component. Available in Worker version 2.16 and later.
: **Available for**: Sources.
### Buffer

Use these metrics to analyze buffer performance. All metrics are emitted on a one-second interval, unless otherwise stated.

{{% observability_pipelines/metrics/buffer/processors %}}

{{% /tab %}}
{{% tab "Destinations" %}}

### Throughput

Bytes out
: **Metric**: `pipelines.component_sent_bytes_total`
: **Description**: The number of raw bytes written to the destination's output, after encoding and transformations.

Events in
: **Metric**: `pipelines.component_received_events_total`
: **Description**: The number of events received by the component.

Events out
: **Metric**: `pipelines.component_sent_events_total`
: **Description**: The number of events the component sends downstream.

Event bytes in
: **Metric**: `pipelines.component_received_event_bytes_total`
: **Description**: The byte size of events received by the component.

Event bytes out
: **Metric**: `pipelines.component_sent_event_bytes_total`
: **Description**: The byte size of events the component sends downstream.

### Errors and data dropped

Errors
: **Metric**: `pipelines.component_errors_total`
: **Description**: The number of errors encountered by the component. Depending on the component, this metric can include an `error_code`, `error_type`, or `reason` tag that describes the error.

Data dropped intentionally or unintentionally
: **Metric**: `pipelines.component_discarded_events_total`
: **Description**: The number of events dropped. **Note**: To break down this metric, use the `intentional:true` tag to filter for events that are intentionally dropped or the `intentional:false` tag for events that are not intentionally dropped.

Check notice on line 308 in content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.sentencelength

Suggestion: Try to keep your sentence length to 25 words or fewer.

### Performance

Utilization
: **Metric**: `pipelines.utilization`
: **Description**: The component's activity. A value of `0` indicates an idle component that is waiting for input. A value close to `1` indicates a component that is never idle, meaning that the component is likely a bottleneck in the processing topology that is creating backpressure. This might cause events to be dropped.

Check notice on line 314 in content/en/observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.sentencelength

Suggestion: Try to keep your sentence length to 25 words or fewer.

### Buffer

Use these metrics to analyze buffer performance. All metrics are emitted on a one-second interval, unless otherwise stated.

{{% observability_pipelines/metrics/buffer/destinations %}}

#### Deprecated buffer metrics

{{% observability_pipelines/metrics/buffer/deprecated_destination_metrics %}}

{{% /tab %}}
{{< /tabs >}}

## HTTP server metrics

Expand Down Expand Up @@ -339,26 +429,6 @@
: **Description**: The historical mean RTT, in seconds, for HTTP requests to this destination, used as the long-term baseline for adaptive concurrency adjustments.
: **Metric type**: distribution

## Buffer metrics

Use these metrics to analyze buffer performance. All metrics are emitted on a one-second interval, unless otherwise stated.

### Source buffer metrics

{{% observability_pipelines/metrics/buffer/sources %}}

### Processor buffer metrics

{{% observability_pipelines/metrics/buffer/processors %}}

### Destination buffer metrics

{{% observability_pipelines/metrics/buffer/destinations %}}

### Deprecated buffer metrics

{{% observability_pipelines/metrics/buffer/deprecated_destination_metrics %}}

## Further reading

{{< partial name="whats-next/whats-next.html" >}}
Expand Down
Loading