SubmitSamples batches with for i := 0; i < len(samples); i += telemetry.MaxDeviceLatencySamplesPerBatch and returns on the first failing batch (controlplane/telemetry/internal/telemetry/submitter.go:95-170). Tick then re-invokes it from scratch for up to MaxAttempts, so any batch that already committed is written again.
Duplicates inflate next_sample_index, and because the account carries one start_timestamp_microseconds and one sampling_interval_microseconds rather than a timestamp per sample, that shifts every synthesized timestamp for the account. Enough of them can also push it to SamplesAccountFull, where the submitter drops the whole partition.
This is pre-existing, but #4143 makes it reachable at scale. Before that change the pinger stopped adding during a ledger outage, so tmp was normally 6 samples — one batch, no partial-failure window. With the cached-epoch fallback an hours-long backlog is ~17 sequential transactions per attempt, retried up to 5 times, precisely while the RPC is still flaky.
Fix: track the batch index across attempts, or retry per batch rather than per call.
Reported by @ben-dz in review of #4143.
SubmitSamplesbatches withfor i := 0; i < len(samples); i += telemetry.MaxDeviceLatencySamplesPerBatchand returns on the first failing batch (controlplane/telemetry/internal/telemetry/submitter.go:95-170).Tickthen re-invokes it from scratch for up toMaxAttempts, so any batch that already committed is written again.Duplicates inflate
next_sample_index, and because the account carries onestart_timestamp_microsecondsand onesampling_interval_microsecondsrather than a timestamp per sample, that shifts every synthesized timestamp for the account. Enough of them can also push it toSamplesAccountFull, where the submitter drops the whole partition.This is pre-existing, but #4143 makes it reachable at scale. Before that change the pinger stopped adding during a ledger outage, so
tmpwas normally 6 samples — one batch, no partial-failure window. With the cached-epoch fallback an hours-long backlog is ~17 sequential transactions per attempt, retried up to 5 times, precisely while the RPC is still flaky.Fix: track the batch index across attempts, or retry per batch rather than per call.
Reported by @ben-dz in review of #4143.