RFC: Multi-blob add - #29
hannahhoward wants to merge 7 commits into
Conversation
Batch upload protocol: /blob/batch/add plans placement for M blobs in one invocation and re-delegates the node verbs to the agent under policy-bound delegations; /blob/batch/commit relays node-signed accept receipts. Two upload-service round trips per batch, two per node, no node changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UycQw45qWtuVKmvrzZVEKa
The batch pair concludes with /ucan/conclude generalized to many receipts: per winner, the node-signed /blob/allocate receipt plus a client-minted /http/put receipt. Allocate-only re-delegation; accept stays sprue-invoked behind a durable conclusion journal; today's add->allocate->put->accept chain shape is preserved. Applies the verification pass: corrected current-flow counts, conclude authorization stated as a strengthening, placement replica bound, no-freshness delegation check at conclude, quota holds, and house-format sections. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UycQw45qWtuVKmvrzZVEKa
Agent drives both node verbs under placement delegations; the client completes at the accept round with no polling and one synchronous sprue exchange. Registration rides /ucan/conclude as open receipt delivery (node SHOULD conclude, agent SHOULD deliver; replica-transfer precedent), replacing the conclusion journal. Soundness additions from the adversarial pass: a delegation-expiry-derived registration horizon with matching node-side reclaim floor, cause-keyed and pending tombstones with per-(digest,space) serialization, batch-support advertisement gating placement, replica count in delegation metadata, and quota-debit rules. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UycQw45qWtuVKmvrzZVEKa
Registration becomes a chain-visible virtual task in the /http/put tradition: sprue mints it per winning acceptance inside the /blob/accept conclusion handler, no-nonce so concurrent deliveries converge on one receipt. ConcludeOutcome carries a registration link instead of a boolean; Surplus outcomes link the registered acceptances; removal retires the register tasks via the cause-keyed tombstone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UycQw45qWtuVKmvrzZVEKa
The batch-add response returns to the deployed container contract: per candidate, the promise-chained task set (allocate, put, accept, register), sprue-signed, no-nonce, window expiries — the agent relays rather than authors. /blob/reject stays the one key-bound delegation (policy pinning space+digest). Conclusion verification collapses to sprue's own signatures with the register task as batch marker and replica carrier. Adds /blob/batch/abort (cause termination + pending tombstones), the three blob-removal amendments stated in Compatibility, node-side refusal of register-less accepts, deferred PDP enqueue until registration signal, and the delegation model as Alternatives considered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UycQw45qWtuVKmvrzZVEKa
Verified against upstream Storj: over-provisioning is redundancy-space only (110 limits for 80 needed, one node per piece, no spares; replacement via an amendment RPC). The RFC drops the candidate-set/racing machinery accordingly: one node, one program per blob — the current protocol per blob, batched. Sharding clients over-provision in their own blob set and walk away from the tail; extend is the amendment analogue with cumulative client-supplied exclusions. Also removes the replicas field (bound is one; replication moved to Future work), collapses to a single expiry (window + grace = horizon), and scopes abort by digest so a tail can be renounced while its batch stays live. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UycQw45qWtuVKmvrzZVEKa
Drops the /blob/reject delegation — the last non-invocation artifact in the placement. Node-side cleanup becomes sprue's translation, exactly as /blob/abort maps to /blob/reject in the removal RFC: batch abort writes the tombstones then forwards rejects to every renounced placement's provider, best-effort, routed from the stored batch-add receipt. The response container now holds only bearer-safe tasks; nothing a holder can do is destructive. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UycQw45qWtuVKmvrzZVEKa
|
Drive by comment - just wondering if it's better for the batch operations to be suffixes to the existing commands i.e. a Since it's possible to achieve the same result as a batch add by just invoking |
|
Another thought - batch remove? |
| receipt Link # the delivered receipt | ||
| registration optional Link # the /blob/register receipt, in the response container | ||
| reason optional String # named outcome when this delivery registered nothing, | ||
| # e.g. Surplus, Removed, Expired, InvalidReceipt, UnknownPlacement |
There was a problem hiding this comment.
I'm not sure I agree with this. This is tri-directional and ties a specific workflow into something that is designed to be generic.
I feel like the /blob/register task should be a promise in the receipt that was sent to /ucan/conclude and so could just be included as a receipt in the response container - i.e. you already know the CID of task that was run so no need to specify it again here. ConcludeOK should always be a unit.
|
BTW, I do love this! |
bajtos
left a comment
bajtos
left a comment
There was a problem hiding this comment.
I love the proposed direction 👏🏻
I left a bunch of review comments below. At the high level:
- We already modified
/ucan/concludeto accept a batch of items, see fil-forge/ucan-protocol-specs#7. The proposal needs to reflect that. - For a typical S3 client, we need to optimise the performance of single-blob file uploads + many-blob concludes, because
aws s3splits large files into 8MB chunks and uploads them as individual parts. This proposal is focused on improving a different path. However, it introduces changes & primitives that will help for the multi-part upload performance too! - I'd like to discuss whether we really need and want to preserve backwards compatibility.
|
|
||
| ## Motivation | ||
|
|
||
| Uploading one blob today costs the client two POSTs to the upload service, an HTTP PUT of the bytes to the storage node, and a receipt poll, and costs the upload service two invocations against the storage node, all serial: `/blob/add`, the PUT, `/ucan/conclude` carrying a self-issued `/http/put` receipt, then `GET /receipt/{cid}` until the `/blob/accept` receipt appears — at least one more round trip per blob, and a second of added latency whenever the receipt is not ready on the first try. An object of M blobs pays that chain M times. Client-side parallelism hides some of the latency but none of the message count. |
There was a problem hiding this comment.
FWIW, we have recently upgraded /ucan/conclude to send an array of blobs, see fil-forge/ucan-protocol-specs#7
|
|
||
| This RFC proposes a batch upload protocol: | ||
|
|
||
| - `/blob/batch/add` — announce M blobs in one invocation. The upload service plans placement for the whole batch in one pass and returns, per blob, one placement carrying the upload's tasks as pre-minted, promise-chained invocations — `/blob/allocate`, `/http/put`, `/blob/accept`, `/blob/register` — exactly as today's `/blob/add` response carries them for one blob and one node. |
There was a problem hiding this comment.
Let's discuss the operation name.
If I recall and understand correctly how UCAN permissions work, the operation /blob/batch/add requires a new permission. Clients with /blob/add permission only are not allowed to invoke this operation.
What if we call this operation /blob/add/batch, would that allow all clients with /blob/add permissions to also invoke /blob/add/batch, since it's a nested path?
| This RFC proposes a batch upload protocol: | ||
|
|
||
| - `/blob/batch/add` — announce M blobs in one invocation. The upload service plans placement for the whole batch in one pass and returns, per blob, one placement carrying the upload's tasks as pre-minted, promise-chained invocations — `/blob/allocate`, `/http/put`, `/blob/accept`, `/blob/register` — exactly as today's `/blob/add` response carries them for one blob and one node. | ||
| - The agent drives the node exchange itself: one container of allocate invocations per node, the PUTs in parallel, one container of accept invocations per node. The accept receipts — location claim and PDP promise included — are in the agent's hands when the accept round completes. Nothing is polled and nothing is awaited from the upload service. A sharding client over-provisions in its blob set — announce more pieces than the code needs, finish enough, walk away from the tail — which needs no protocol support beyond batching itself. |
There was a problem hiding this comment.
What is "the agent" here? Is it Ingot/Guppy?
There was a problem hiding this comment.
I see that "agent" is defined in "Roles" below as "Ingot". I propose moving the "Roles" section to the top of the document to define terms before they are used in the text.
|
|
||
| - `/blob/batch/add` — announce M blobs in one invocation. The upload service plans placement for the whole batch in one pass and returns, per blob, one placement carrying the upload's tasks as pre-minted, promise-chained invocations — `/blob/allocate`, `/http/put`, `/blob/accept`, `/blob/register` — exactly as today's `/blob/add` response carries them for one blob and one node. | ||
| - The agent drives the node exchange itself: one container of allocate invocations per node, the PUTs in parallel, one container of accept invocations per node. The accept receipts — location claim and PDP promise included — are in the agent's hands when the accept round completes. Nothing is polled and nothing is awaited from the upload service. A sharding client over-provisions in its blob set — announce more pieces than the code needs, finish enough, walk away from the tail — which needs no protocol support beyond batching itself. | ||
| - Registration travels as receipt delivery over `/ucan/conclude`, which stays what it is today: an open conclusion mechanism that anyone holding a receipt may invoke. The storage node SHOULD conclude each accept it executes; the agent SHOULD deliver its collected receipts as well, batched. The upload service executes the pre-minted `/blob/register` task idempotently from whichever delivery arrives first. |
There was a problem hiding this comment.
What is "the storage node"? Is it Piri or Ingot?
There was a problem hiding this comment.
Found it: the section "Roles" below defines Piri as "a Forge network storage node".
I propose moving the "Roles" section to the top of the document to define terms before they are used in the text.
| - `/blob/batch/add` — announce M blobs in one invocation. The upload service plans placement for the whole batch in one pass and returns, per blob, one placement carrying the upload's tasks as pre-minted, promise-chained invocations — `/blob/allocate`, `/http/put`, `/blob/accept`, `/blob/register` — exactly as today's `/blob/add` response carries them for one blob and one node. | ||
| - The agent drives the node exchange itself: one container of allocate invocations per node, the PUTs in parallel, one container of accept invocations per node. The accept receipts — location claim and PDP promise included — are in the agent's hands when the accept round completes. Nothing is polled and nothing is awaited from the upload service. A sharding client over-provisions in its blob set — announce more pieces than the code needs, finish enough, walk away from the tail — which needs no protocol support beyond batching itself. | ||
| - Registration travels as receipt delivery over `/ucan/conclude`, which stays what it is today: an open conclusion mechanism that anyone holding a receipt may invoke. The storage node SHOULD conclude each accept it executes; the agent SHOULD deliver its collected receipts as well, batched. The upload service executes the pre-minted `/blob/register` task idempotently from whichever delivery arrives first. | ||
| - `/blob/batch/extend` — a replacement placement when a blob's node fails; `/blob/batch/abort` — cancel a batch's outstanding placements. |
There was a problem hiding this comment.
Do we need to introduce the new "batch" versions of operations? Cannot we keep existing operation names and change the payloads from "one item" to "an array of items", similarly to what we did in fil-forge/ucan-protocol-specs#7?
It's a breaking change, but IMO we can afford that, since we are not in production and not v1.0 yet.
| ```ipldsch | ||
| type BatchAbortOK struct { | ||
| cancelled [Bytes] # digests whose placements are tombstoned | ||
| registered [Bytes] # digests already registered; use /blob/remove |
There was a problem hiding this comment.
Do we have a batch version of /blob/remove so that the service receiving BatchAbortOK can remove all blobs in as few requests as possible?
|
|
||
| * Issuer: Sprue | ||
| * Audience: Piri | ||
| * Subject: The provider |
There was a problem hiding this comment.
What is "The provider"? This is not defined in the Roles section at the top.
| ``` | ||
| </details> | ||
|
|
||
| ### The node legs: `/blob/allocate` and `/blob/accept`, unchanged verbs |
There was a problem hiding this comment.
When completing a multi-part upload request, we need to call /blob/accept for potentially thousands of blobs at once. With one invocation per block, this step is a performance bottleneck.
I propose reworking /blob/accept to accept an array of blobs, similarly to how we reworked /blob/conclude in fil-forge/ucan-protocol-specs#7.
There was a problem hiding this comment.
This is not needed, we are already sending multiple /blob/accept invocations in a single HTTP request, see https://github.com/fil-forge/sprue/blob/506f5f68a8064967cfe13c83fcbef12f5fdafe4a/pkg/piriclient/client.go#L195-L252
|
|
||
| The residual window is bounded: a blob is accepted-but-unregistered until the first delivery lands, and if every holder crashes before delivering, until one recovers or the horizon closes it out. | ||
|
|
||
| The client's completion does not wait for any of this. Resolvability comes from accept itself (the node publishes its location claim synchronously, as today), so the S3 200 returns on accept receipts; registration — billing, retain seeding, repair and removal routing — converges with delivery, typically within the agent's own immediate conclude. A client that wants a stronger signal than the 200 holds one artifact: the `/blob/register` receipt, addressable by task CID from the batch-add response, verifiable offline and presentable to anyone. |
There was a problem hiding this comment.
IIUC, our S3 gateway (Ingot) returns 200 OK before the storage node (Piri) persists the blob.
How will we recover if Ingot's machine crashes during that window and loses all locally spooled bytes? We already told the client that the data was persisted, so we don't have a way to indicate an error and to ask the client to retry (re-upload).
IMO, Ingot should wait until the sufficient number of storage nodes confirmed the blob registration, before it returns 200 OK to the client.
What am I missing?
There was a problem hiding this comment.
I think my comment is pointing at the open question listed at the bottom: "Whether the agent awaits its conclude response"
|
|
||
| - **Storage nodes**: the verbs, argument types, results and issuer are unchanged — nodes validate today's exact chain; only the courier differs. Outbound conclusion has an existing precedent: piri already delivers replica-transfer receipts via `/ucan/conclude`. What is new is opt-in: a node advertises batch support at registration, taking on the delivery and reclaim behaviors, and sprue places batches only on nodes that have. Nodes that never advertise keep serving the legacy flow exactly as today and see no batch traffic, so the delivery model's guarantees never depend on a node that hasn't signed up for them. | ||
| - **`/blob/add` and the [blob protocol spec](https://github.com/fil-forge/ucan-protocol-specs/blob/main/blob.md)**: the single-blob flow remains, unchanged, for callers that cannot know digests up front, and the spec's response-container contract generalizes rather than changes — the container still carries the upload's promise-chained task invocations; there are simply more of them, plus the explicit terminal register task. A streaming client MAY batch per flush window — announce the blobs whose digests it has, upload, accept, deliver, repeat. | ||
| - **`/ucan/conclude`**: the plural field is additive, and the `/blob/accept` conclusion handler is a new entry in the existing dispatch map; existing single-receipt clients and the legacy `/http/put` handler are untouched. |
There was a problem hiding this comment.
We already shipped a breaking change to accept only the plural field. We need to figure out if and how we want to support both old-style conclude requests, where Sprue calls Piri, and the new style, where receipts are already included in the conclude request.
3 participants
Summary
/blob/batch/addannounces M blobs in one invocation and returns, per blob, one placement carrying the upload as a pre-minted, promise-chained program — sprue-signed/blob/allocate,/http/put,/blob/accept,/blob/registerinvocations. Per blob it is exactly the current protocol's container contract; batching is the only change. The agent relays the tasks and finishes with one synchronous sprue exchange and zero polling. A sharding client over-provisions in its own blob set (announce more pieces than needed, finish enough, walk away from the tail) — no protocol support required beyond batching./ucan/concludeas an open receipt-delivery mechanism: the node SHOULD conclude each accept, the agent SHOULD deliver batched, and sprue executes the pre-minted/blob/registertask idempotently — verification is sprue checking its own signatures, bounded by a registration horizon (task expiry + grace) and tombstone rules, with registration a chain-visible sprue-signed receipt./blob/batch/extendre-places failures with cumulative exclusions;/blob/batch/abortcancels a batch or named digests and forwards/blob/rejectto the affected nodes, exactly as the removal RFC's abort translates today.Read the docs rendered
🤖 Generated with Claude Code
https://claude.ai/code/session_01UycQw45qWtuVKmvrzZVEKa