Multipart upload: single-file uploads are capped at 5 GB and fail late #978

The limit

ndi.cloud.uploadSingleFile gets one presigned URL from ndi.cloud.api.files.getFileUploadURL and hands it to ndi.cloud.api.files.putFiles, which does a single HTTP PUT. S3 caps a single PUT at 5 GB; past that it answers EntityTooLarge.

Nothing catches it first. There is no multipart support and no size guard in +ndi/+cloud/, so an oversized file fails late — after minutes of transfer — with an opaque 400, and starts from zero on retry. putFiles' curl path (curl -fsSL -X PUT --upload-file) cannot do multipart either.

The use case that raised it

Exporting a bin1-only .gef from a pyramid, so a section can be handed to someone as a file SAW-family tools read, without the ~40 GB of extra bin levels the vendor file carries.

Measured on a real ferret hemisphere (Y00709C1E2.tissue.gef, 48.32 GB, 771,217,356 records at bin1, 26,444 genes):

So the very first file this feature wants to produce lands above the cap and below any plausible workaround. Deeper sequencing scales it linearly.

What is NOT affected

The document/binary upload path is fine and should stay as it is. ndi.cloud.upload.zipForUpload batches into zips at a 25 MB default before calling putFiles, so it never approaches the limit. This issue is only about uploadSingleFile and any future caller handing over one large file.

Scoping

Server API

The client only ever receives presigned URLs, so every step of an S3 multipart upload needs an endpoint. Four required, one optional but worth having:

The last one is what makes resume possible rather than just large uploads, and resume is arguably the bigger win: an 8 GB transfer that dies at 90% currently costs the whole thing.

Two server-side decisions worth settling early rather than discovering: whether part URLs are minted in bulk or one at a time (bulk is far fewer round trips for a 1,600-part upload at 5 MB parts), and how long abandoned multipart uploads are allowed to linger — S3 bills for uncompleted parts until they are aborted, so a lifecycle rule is part of the work, not an afterthought.

ndi.cloud.api.files.*

New, following the naming already in the package:

• createMultipartUpload.m — begin, return uploadId
• getMultipartPartURLs.m — presigned URLs for a range of part numbers
• completeMultipartUpload.m — send the ordered (partNumber, ETag) list
• abortMultipartUpload.m — cleanup, and onCleanup in the caller so a Ctrl-C does not strand parts
• listMultipartParts.m — for resume

Changed:

• putFiles.m — route to multipart above a threshold, keeping the single PUT below it. The signature can stay; the choice belongs inside rather than at every call site.
• +api/+implementation/+files/PutFiles.m — needs a part-sized sibling, PutFilePart.m.
• uploadSingleFile.m — mostly unchanged if putFiles absorbs the branch, but it should surface progress: a multi-GB upload with no feedback is indistinguishable from a hang, which is the same complaint the GEF ingest just fixed.

The one that will bite

CompleteMultipartUpload needs each part's ETag, taken from the PUT response header. That splits the two implementations in PutFiles.m:

• the matlab.net.http.RequestMessage path returns a ResponseMessage and can read ETag directly;
• the curl path — which is the default (useCurl = true) — discards headers under -fsSL. It would need -D - or -w '%{header_json}' and then parsing.

Worth deciding up front whether multipart uses the MATLAB HTTP path only, or whether the curl path is taught to capture headers. Silently getting an empty ETag would produce a completed upload with corrupt parts, which is the worst available failure.

Interim

A size check in putFiles that refuses anything over 5 GB, naming the limit and pointing here. One second and a clear message instead of ten minutes and a 400. Small enough to do independently of the rest.

Related

• The same shape exists in NDI-python: ndi/cloud/api/files.py::putFiles is a single requests.put, with no multipart and no guard. putFileBytes there is worse — it holds the whole payload in memory — though the file path does not use it.
• The exporter that motivates this is not yet written; it is planned to sit with buildOpossumDocuments in bscholl-genomics-python/ndiMatlabIntegration/. Whether the exported .gef ever needs to reach the cloud at all is still open — if it stays a local deliverable, this becomes lower priority rather than a blocker.