Skip to content

feat: Replace multipart encoder dependency with a streaming encoder - #1596

Closed
congminh1254 wants to merge 3 commits into
mainfrom
codegen-release-remove-requests-toolbelt
Closed

congminh1254 wants to merge 3 commits into
mainfrom
codegen-release-remove-requests-toolbelt

Conversation

@congminh1254

@congminh1254 congminh1254 commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

Summary

  • This PR adds box_sdk_gen/networking/multipart_stream.py, which contains MultipartStream. It is a lazy, file-like multipart/form-data body built on urllib3's RequestField formatter, so it produces the same header formatting as toolbelt and requests files=.
  • When the stream sizes are known, it sets an exact Content-Length. When a stream isn't seekable, it falls back to chunked transfer encoding.
  • If a stream ends before its declared size, it fails fast.
  • If a stream grows after the size was computed, it sends only the declared size.
  • BoxNetworkClient now builds the multipart body with MultipartStream. The requests-toolbelt dependency is removed; install_requires is now ['requests'].
  • Adds unit tests covering:
    • wire-format parity with urllib3 encode_multipart_formdata
    • a round trip plus retry against a local server
    • chunked uploads for non-seekable streams
    • lazy reads
    • streams that shrink or grow during upload

Branch uses the codegen-release prefix only so CI runs the full tox suite (unit + integration). box_sdk_gen is generated, so this change also needs to land in box-codegen before release.

Validation vs requests-toolbelt

Tested on three environments: Python 3.9, 3.10 and 3.12, with urllib3 1.26 and 2.x, against a local HTTP server.

Wire format. The new body is byte-identical (after normalising the boundary) to toolbelt and to files= in 18 of 25 cases. The other 7 differences are toolbelt bugs that this change fixes:

  • Stream not at the start: toolbelt ignored the stream's current position and sent it from offset 0.
  • Duplicate part names: toolbelt dropped the first part.
  • data={}: toolbelt sent it as an empty file part.
  • Non-seekable input: toolbelt crashed on non-seekable streams, ResponseByteStream, pipes and BufferedReader. These now stream using chunked encoding.

Other behaviour changes:

  • File truncated mid-upload: toolbelt busy-loops forever at 100% CPU. The new encoder fails in about 14 ms with a clear error.
  • Retrying a non-seekable stream: instead of a raw UnsupportedOperation, the caller now gets the SDK's "cannot be retried" error.
  • Unchanged: retries, 401 re-auth, timeouts, and upload_file, upload_file_version and create_user_avatar behave the same as before.

Performance:

case toolbelt new
500 MB upload wall time (py3.12) 1.47 s 0.96 s
8×50 MB concurrent uploads wall time (py3.12) 2.78 s 0.95 s
Peak memory flat flat (0 MB)
Client CPU per small request 64 µs 40 µs

For comparison, requests files= needs 1000 MB of RSS for a 500 MB upload.

Security:

  • The boundary comes from urllib3 choose_boundary, which uses os.urandom.
  • CRLF in the filename or part name is escaped, the same as before.
  • The dependency surface shrinks by about 3.8k lines of code.

Non-seekable streams

Uploads from non-seekable streams now use Transfer-Encoding: chunked; previously toolbelt crashed on them. test/multipart_uploads.py uploads a 5 MB file and a 1 MB file version from a non-seekable stream to Box, then checks the size and the downloaded bytes. It passes in CI, so the Box upload endpoints accept chunked multipart bodies. These uploads still can't be retried: a retry raises the SDK's "cannot be retried" error.

Follow-up hardening (second commit)

  • A stream positioned past its end is sent as an empty part. Before, this produced a negative size.
  • Streams whose seek() returns None are now supported.
  • A part stream that ends before its declared size now fails immediately with BoxSDKError, instead of being retried.
  • urllib3 is listed in install_requires, because the SDK now imports it directly.

Known issue

  • If a content_type contains CRLF, it is still not escaped. This is a pre-existing issue, and toolbelt had it too.

Test plan

  • pytest test/box_network_client.py: 78 tests pass locally
  • CI: full tox (unit + integration) on Python 3.8 and 3.13, 240 passed
  • pycodestyle and pylint (not run by CI)

🤖 Generated with Claude Code

@congminh1254
congminh1254 requested a review from a team September 28, 2026 09:52
@congminh1254 congminh1254 changed the title feat: Replace requests-toolbelt with streaming MultipartStream encoder feat: Replace toolbelt multipart encoder with a streaming encoder Sep 28, 2026
Add a small file-like multipart/form-data body (MultipartStream) built on
urllib3's RequestField formatter. It reads part streams lazily, sends an
exact Content-Length for seekable streams and falls back to chunked
transfer encoding for non-seekable ones. This removes the
requests-toolbelt dependency.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@congminh1254 congminh1254 changed the title feat: Replace toolbelt multipart encoder with a streaming encoder feat: Replace multipart encoder dependency with a streaming encoder Sep 28, 2026
@congminh1254
congminh1254 force-pushed the codegen-release-remove-requests-toolbelt branch from 87fbf9a to b1a934e Compare September 28, 2026 09:57
- Clamp the part size to zero for streams positioned past their end.
- Read the end position with tell() so streams whose seek() returns None work.
- Fail immediately with BoxSDKError when a part stream ends before its
  declared size, instead of retrying a request that can't succeed.
- List urllib3 in install_requires, since it is now imported directly.
- Add an integration test uploading a file and a file version from a
  non-seekable stream, which is sent with chunked transfer encoding.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@socket-security

socket-security Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedpypi/​urllib3@​2.8.097100100100100

View full report

arjankowski
arjankowski previously approved these changes Sep 28, 2026

@arjankowski arjankowski left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

based on my python knowledge, the looks good.
I've verified this on my end and it works fine

Comment on lines +380 to +381
assert isinstance(api_request.data, MultipartStream)
assert mock_byte_stream.tell() == 0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: This test lost its field-level assertions, so a regression in how attributes is serialized or ordered no longer fails here. Could we assert on the rendered body?

mwwoda
mwwoda previously approved these changes Sep 28, 2026
Check the exact body so a regression in attributes serialization, part
order or file part headers fails the test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@congminh1254
congminh1254 dismissed stale reviews from mwwoda and arjankowski via 96726c1 September 28, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants