perf: concurrent PutObject parts

[alanshaw]

Second attempt refs #140

[alanshaw]

/forge-perf

[alanshaw]

/forge-perf

[github-actions]

forge-perf

Commit	34d0b4fa2be27c7ee06a03e57db95e94690dc45b
Image	ghcr.io/fil-forge/piri:pr-148-34d0b4f (sha256:0a24a49da040)
Pairs	1 (main, branch)
State	Done: slower
Pairing	exp-piri-pr148-34d0b4fa2be2-36699209201
Logs	workflow run

Run	Role	Class	Median	p5	Traces
main-20260930t100236z	main	valid (few_windows, nic_allowance_exceeded, traced)	1.94 GB/s	1.84 GB/s	Grafana
main-20260930t101113z	branch	valid (few_windows, nic_allowance_exceeded, traced)	1.79 GB/s	1.59 GB/s	Grafana

	Median	p5
Branch against main	-7.7%	-13.4%
Noise band	±3.5%	±11.0%

Verdict: slower, judged on the median against its noise band. Positive deltas mean the branch ingests faster.

Main runs the current main set; branch swaps in this commit's image. The runs table on the forge-perf page lists each run.

[alanshaw]

Well, this sucks

[alanshaw]

The trace answers it. The concurrency is working exactly as coded; it just can't help, because the time is spent waiting on S3, and adding connections makes the S3 side worse.

What the trace shows for this upload. One 130.6 MiB blob into Piri, nine parts. Piri's object store put took 3.39 s. The ingest side is fast: whenever a buffer was free, the next 16 MiB part was read from Ingot in about 40 to 60 ms, so the whole body could have been read in about half a second. Every part was dispatched by 1.34 s into the put. The put then sat idle for 1.95 s waiting for the slowest in-flight part, and only then sent CompleteMultipartUpload.

The part PUTs to S3 are the problem. Same size, wildly different times:

Part start (ms)	PUT duration (ms)
2139	292
2376	2009
2434	349
2477	900
2496	336
2847	2478
2851	445
3319	620
3380	158 (the 2.6 MiB tail part)

No retries: each part has exactly one HTTP span with a 200. Two parts took 2.0 and 2.5 s while their neighbours, dispatched at the same moment, took 300 to 450 ms. That is a bimodal S3 tail, the signature of connections stalling on packet loss at the instance's egress cap rather than S3 being uniformly slow.

Why parallel parts don't help here. This run was on the m9gd.8xlarge with 64 drill workers, which the calibration sweep already put at the box's S3 ceiling. Each upload's bytes leave the box at the aggregate rate divided by 64 regardless of how many connections it opens. With the PR, 64 uploads each hold up to 4 parts in flight, so 256 connections share the NIC instead of 64. Each connection gets a quarter of the bandwidth, the ENA allowance drops more packets, and stalled connections crawl. An upload can't finish until its slowest part lands, so per-upload latency tracks the worst of its four connections instead of the average of one. Two more costs come on top: minio-go keeps only 16 idle connections per host, so most parts dial a new TLS connection and re-run slow start, and every put allocates and zeroes a 64 MiB buffer pool.

What it means for the PR. The read-ahead in the parallel path also gates on the same four buffers, so from part 5 onward every read waited for an upload to finish. The gaps between part starts match the end times of earlier parts exactly. In other words the parallel path turned a fast reader into one paced by S3 tail latency, with no aggregate gain. Per-upload concurrency only pays when a single upload has the box's bandwidth to itself, which the drill never does.

If the goal is faster ingest under the drill, the knob is the aggregate: fewer, fatter connections, or getting the box's S3 egress higher (the s3-ceiling numbers), not more connections per upload. I'd close this PR and leave the default options, which also keeps the s3-ceiling calibration modelling the real call.

[alanshaw]

What would you recommend in order to speed up the ingest then?

Stop tuning the S3 client. The forge-perf numbers say main is already at the box's network ceiling, so no Piri-side upload change can show up in the drill.

m9gd.8xlarge	GB/s
NIC baseline	2.13
S3 PUT ceiling (calibrated 2026-09-27, minio-go defaults)	2.08 p5 / 2.10 median
main, the paired run for PR 148	1.84 p5 / 1.94 median
PR 148	1.59 p5 / 1.79 median

Main runs at 93 percent of the S3 ceiling and 91 percent of the NIC, and both runs carry the nic_allowance_exceeded flag. Every ingested byte leaves the box exactly once, to S3, so aggregate ingest cannot exceed egress. The same holds on tier 1, where the drill's noise band sits at or above the 0.53 GB/s NIC baseline. The 7 percent gap to the ceiling is the time workers spend not pushing to S3: Ingot's spool, the Sprue round trips, allocate and accept. That gap is the whole prize on this instrument, and it is smaller than the run-to-run noise band.

What I would do instead, in order:

• Finish the trailer-hash work. It is the one change that attacks the actual serial cost. In the trace this request spent 1.7 s spooling in Ingot and then 3.4 s uploading through Piri, back to back. Overlapping them cuts per-request latency by roughly the spool time and keeps each worker on the NIC a larger share of the time. On the drill that closes most of the 7 percent. For a real client with its own NIC it is the difference between 5.8 s and about 3.5 s per 130 MiB object, which is the number users feel.
• Leave Piri's PutObject at minio-go defaults and close PR 148. The sequential path is what the s3-ceiling tool models, and the 4 percent of initiate plus complete overhead is not worth a divergence from the calibration.
• If you want a bigger drill number, change the box. The 16xlarge is calibrated at 4.19 GB/s S3 and 4.74 GB/s NVMe. Ingest on this design scales with egress, and that is a tier decision, not a code change.
• If the goal is production rather than the drill, measure the production shape before optimising for it. Production Piri writes to MinIO on local NVMe, where the ceiling is disk and CPU rather than a NIC allowance, and the S3-over-TLS tuning here says nothing about that path.

One small thing worth a look on the Piri side while the rest lands: in the accept step the HEAD on the freshly written blob took 233 ms, which is on the critical path of every conclude. It is a symptom of the saturated NIC rather than a bug, but if allocate already records the size, that HEAD can go.

[hannahhoward]

/forge-perf

[github-actions]

forge-perf

Commit	34d0b4fa2be27c7ee06a03e57db95e94690dc45b
Image	ghcr.io/fil-forge/piri:pr-148-34d0b4f (sha256:eb1cc4f035fa)
Pairs	1 (main, branch)
State	Done: within noise
Pairing	exp-piri-pr148-34d0b4fa2be2-36938581235
Logs	workflow run

Run	Role	Class	Median	p5	Traces
main-20261001t230921z	main	valid (nic_allowance_exceeded, traced)	2.92 GB/s	2.86 GB/s	Grafana
main-20261001t232807z	branch	valid (nic_allowance_exceeded, traced)	2.97 GB/s	2.49 GB/s	Grafana

	Median	p5
Branch against main	+1.7%	-13.2%
Noise band	±3.5%	±11.0%

Verdict: within noise, judged on the median against its noise band. Positive deltas mean the branch ingests faster.

Main runs the current main set; branch swaps in this commit's image. The runs table on the forge-perf page lists each run.