Repository navigation
Add Python pickle buffer support #1080
Description
Currently when pickling an IMP C++ object we serialize into a C++ std::string and then pass that back to Python as a bytes object, which Python can then pickle. This creates several copies of the underlying data. Consider adding support for pickle protocol version 5 if available (Python >= 3.8, or the backported pickle5 module in Python >= 3.5) as per PEP 574 to store the data in a Python buffer. This should in principle use less memory and be faster, particularly for large objects such as IMP::em::DensityMap or IMP::Model. For example, see the work done for NumPy arrays at numpy/numpy#12011.
Activity
Avoid extra copies when pickling and unpickling
Serialize DensityMap data as a single block
aozalevsky commentedon Oct 17, 2026
#1127 (merged) addresses the copies:
_get_as_binary()now copies the serialized data once, straight into thebytesobject, and_set_from_binary()reads it directly from thebytesobject.DensityMapalso writes its voxel data as one block, which makes pickling a 256³ map about 2.5x faster (unpickling 2x). The pickle format is unchanged.Protocol 5 out-of-band buffers are not done yet. Notes for whoever picks them up:
DensityMap,Model). A prototype that wrapped every object's state in aPickleBufferfrom a Python__reduce_ex__made pickling 100,000Vector3Ds with protocol 5 slower (0.16 s to 0.26 s).Py_buffer, which is only in the limited API from Python 3.11. The Windows installer is built against the 3.10 limited API, so that build would need a copying fallback.RandomNumberGenerator.get_state()/set_state()inIMP_kernel.random.istill use the old copying pattern. The state is about 6.7 KB and is saved once per PMI restart checkpoint, so they were left as is; they could use the same helpers if you'd like them to be consistent.🤖 Generated with Claude Code