Skip to content

Avoid extra copies when pickling, and serialize DensityMap data in one block - #1127

Merged
benmwebb merged 2 commits into
salilab:developfrom
aozalevsky:faster_pickle
Oct 17, 2026
Merged

benmwebb merged 2 commits into
salilab:developfrom
aozalevsky:faster_pickle

Conversation

@aozalevsky

Copy link
Copy Markdown
Contributor

Relates #1080.

Two small changes that make pickling use less memory, and make it faster for density maps, without changing the pickle format:

  1. _get_as_binary() serialized into an ostringstream, then copied the data into a std::string and again into a bytes object; _set_from_binary() copied the bytes into a std::string and again into an istringstream. The data are now copied once, from the stream buffer into the bytes object, and read directly from the bytes object (two small helper classes in swig_helpers_base.h).
  2. DensityMap::serialize() wrote and read the voxel data one value at a time; it now does so with a single cereal::binary_data call.

Pickling a 256³ IMP.em.DensityMap (134 MB) and an IMP.Model with 10⁶ XYZR particles (41 MB); Release build, Python 3.11, median of 3; temporary memory as a multiple of the pickle size:

develop this PR
DensityMap dumps 0.36 s, 2x 0.14 s, 1x
DensityMap loads 0.22 s, 3x 0.11 s, 1x
Model dumps 0.14 s, 2x 0.14 s, 1x
Model loads 0.89 s, 3x 0.89 s, 1x

Pickling 100,000 Vector3Ds is 5-8% faster. Pickles are byte-for-byte the same as before (checked for Model, a restraint, VersionInfo, Vector3D and DensityMap), so old pickles load with this PR and vice versa.

Protocol 5 out-of-band buffers (PEP 574), also suggested in #1080, are left for a follow-up. A prototype that wrapped every object's state in a PickleBuffer from a Python __reduce_ex__ made pickling 100,000 Vector3Ds with protocol 5 slower (0.16 s to 0.26 s), so out-of-band buffers should probably be used only for large objects; and reading them without a copy needs Py_buffer, which is only in the limited API from Python 3.11 (the Windows installer is built against the 3.10 limited API).

Existing tests cover the changed code (e.g. the pickle tests in modules/em/test/test_density_map.py and modules/kernel/test/test_version_info.py). The CI jobs pass locally (Ubuntu 24.04 Release and Debug, Ubuntu 26.04 Fast, clang standing in for macOS), as does a Fast build with IMP_PY_LIMITED_API=3.10, as used for the Windows installer.

🤖 Generated with Claude Code

_get_as_binary() serialized into an ostringstream, copied that into
a std::string and then into a Python bytes object, and
_set_from_binary() copied the bytes into a std::string and then into
an istringstream. Copy straight from the stream buffer into the bytes
object, and read directly from the bytes object instead. Relates salilab#1080.
Write and read the voxel data with one call rather than one per voxel.
The serialized bytes are unchanged. Relates salilab#1080.
@codecov

codecov Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 67.72%. Comparing base (318a8ec) to head (cf988ac).

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1127      +/-   ##
===========================================
- Coverage    67.72%   67.72%   -0.01%     
===========================================
  Files         1811     1811              
  Lines       154809   154809              
  Branches     18519    18520       +1     
===========================================
- Hits        104852   104850       -2     
- Misses       45245    45249       +4     
+ Partials      4712     4710       -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@benmwebb
benmwebb merged commit 9cac172 into salilab:develop Oct 17, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants