feat: Replace multipart encoder dependency with a streaming encoder - #1596
Closed
congminh1254 wants to merge 3 commits into
Closed
congminh1254 wants to merge 3 commits into
congminh1254 wants to merge 3 commits into
Conversation
Add a small file-like multipart/form-data body (MultipartStream) built on urllib3's RequestField formatter. It reads part streams lazily, sends an exact Content-Length for seekable streams and falls back to chunked transfer encoding for non-seekable ones. This removes the requests-toolbelt dependency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
congminh1254
force-pushed
the
codegen-release-remove-requests-toolbelt
branch
from
September 28, 2026 09:57
87fbf9a to
b1a934e
Compare
- Clamp the part size to zero for streams positioned past their end. - Read the end position with tell() so streams whose seek() returns None work. - Fail immediately with BoxSDKError when a part stream ends before its declared size, instead of retrying a request that can't succeed. - List urllib3 in install_requires, since it is now imported directly. - Add an integration test uploading a file and a file version from a non-seekable stream, which is sent with chunked transfer encoding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
arjankowski
previously approved these changes
Sep 28, 2026
arjankowski
left a comment
Contributor
There was a problem hiding this comment.
based on my python knowledge, the looks good.
I've verified this on my end and it works fine
mwwoda
reviewed
Sep 28, 2026
Comment on lines
+380
to
+381
| assert isinstance(api_request.data, MultipartStream) | ||
| assert mock_byte_stream.tell() == 0 |
Contributor
There was a problem hiding this comment.
Nit: This test lost its field-level assertions, so a regression in how attributes is serialized or ordered no longer fails here. Could we assert on the rendered body?
mwwoda
previously approved these changes
Sep 28, 2026
Check the exact body so a regression in attributes serialization, part order or file part headers fails the test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
congminh1254
dismissed stale reviews from mwwoda and arjankowski
via
September 28, 2026 15:01
96726c1
mwwoda
approved these changes
Sep 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
box_sdk_gen/networking/multipart_stream.py, which containsMultipartStream. It is a lazy, file-likemultipart/form-databody built on urllib3'sRequestFieldformatter, so it produces the same header formatting as toolbelt and requestsfiles=.Content-Length. When a stream isn't seekable, it falls back to chunked transfer encoding.BoxNetworkClientnow builds the multipart body withMultipartStream. Therequests-toolbeltdependency is removed;install_requiresis now['requests'].encode_multipart_formdataValidation vs requests-toolbelt
Tested on three environments: Python 3.9, 3.10 and 3.12, with urllib3 1.26 and 2.x, against a local HTTP server.
Wire format. The new body is byte-identical (after normalising the boundary) to toolbelt and to
files=in 18 of 25 cases. The other 7 differences are toolbelt bugs that this change fixes:data={}: toolbelt sent it as an empty file part.ResponseByteStream, pipes andBufferedReader. These now stream using chunked encoding.Other behaviour changes:
UnsupportedOperation, the caller now gets the SDK's "cannot be retried" error.upload_file,upload_file_versionandcreate_user_avatarbehave the same as before.Performance:
For comparison, requests
files=needs 1000 MB of RSS for a 500 MB upload.Security:
choose_boundary, which usesos.urandom.Non-seekable streams
Uploads from non-seekable streams now use
Transfer-Encoding: chunked; previously toolbelt crashed on them.test/multipart_uploads.pyuploads a 5 MB file and a 1 MB file version from a non-seekable stream to Box, then checks the size and the downloaded bytes. It passes in CI, so the Box upload endpoints accept chunked multipart bodies. These uploads still can't be retried: a retry raises the SDK's "cannot be retried" error.Follow-up hardening (second commit)
seek()returns None are now supported.BoxSDKError, instead of being retried.urllib3is listed ininstall_requires, because the SDK now imports it directly.Known issue
content_typecontains CRLF, it is still not escaped. This is a pre-existing issue, and toolbelt had it too.Test plan
pytest test/box_network_client.py: 78 tests pass locally🤖 Generated with Claude Code