TypeError: self._context,self._fmu,self.callBackFunctions,self.callbacks cannot be converted to a Python object for pickling

[jschueller]

Describe the bug
PyFMI model objects cannot be serialized, which makes it impossible from using multiprocessing-like parallel evaluations (joblib etc)

To Reproduce

import pyfmi
import pickle
model = pyfmi.load_fmu("BouncingBall.fmu")
with open("study.pkl", "wb") as f:
    pickle.dump(model, f)
with open("study.pkl", "rb") as f:
    model2 = pickle.load(f)

Observed behavior

model = <pyfmi.fmi2.FMUModelCS2 object at 0x5fe351a1a6d0>
>   ???
E   TypeError: self._context,self._fmu,self.callBackFunctions,self.callbacks cannot be converted to a Python object for pickling

Expected behavior
script passes

Versions

• PyFMI 2.18.3 (latest)
• openmodelica fmu

additional context
what usually causes pain is the cython attributes
maybe a path forward is override getstate/setstate operators, pass pure-python arguments only and recreate a model instance from scratch so that cython attributes are not really copied but reinstanciated

[PeterMeisrimelModelon]

Can you elaborate on exactly what you are trying to do, maybe a complete example? In terms of serializing FMUs, there is the fmiGetFMUState/fmiSetFMUState functionality.

/Peter

[jschueller]

I'm interested in parallelizing fmu simulations with different input values across multiple processes, something like that

import concurrent.futures
import pyfmi

model = pyfmi.load_fmu("BouncingBall.fmu")
X = [1, 2, 3]
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    future_to_x = {executor.submit(model.simulate, {"input": x}): x for x in X}
    for future in concurrent.futures.as_completed(future_to_x):
        x = future_to_x[future]
        y = future.result()

internally Python uses serialization of the context so it fails to pickle the model variable which is an instance of a PyFMI class

Do you have an example of fmiSetFMUState ?
I dont see anything in the doc.
I dont know if it this will help as it should be a pure python object also in order to be serialized.

[PeterMeisrimelModelon]

I have doubts what you are trying to do is going to work this way. I think the safest best is to simply load the FMU N times.

Given e.g., the example of BouncingBall, how would this deal with events? One needs to change the FMU state to appropriate deal with events + the FMU might have internal states that can be affected by events and differ based the external inputs.

On FMUState, first I recommend checking the specification here.

Example on how to use in PyFMI: https://github.com/modelon-community/PyFMI/blob/master/tests/test_fmi3_states.py#L152 (works the same for FMI2)
Note that this is optional functionality, you'll need to check if your FMU generating tool supports generation of FMUs that support this.

Essentially, get_fmu_state creates a Python object housing an opaque pointer of the FMU, that be used to (re)set the FMUs internal state.
de(serialization) allows to convert this internal data structure to a bytes for e.g., external storage.

/Peter

[jschueller]

yes, reloading works
but it would be convenient to do it at the pyfmi level

[jschueller]

@PeterMeisrimelModelon I have a proper fix for this, could you look at #409 ?

[PeterMeisrimelModelon]

Replying here instead of the PR to keep conversation coherent.

I don't see how the PR overcomes the issue with with get/set_fmu_state I described above. The general issue is that

1. If we wanted to add support for pickling FMUModelX classes, it would only really make sense if this works in the general case.
2. The real test would be to simulate and FMU with a proper internal state and then simulate, pickle, unpickle, resume simulation. However, this can only be achieved via get/set_fmu_state (and serialization), because otherwise you are not able to restore the internal FMU state.

[jschueller]

indeed, this doesnt support mid-simulation pickling, and I now added an early return to make it clear - the code throws when trying to serialize an initialized model with an explicit error message:

if self._instantiated_fmu:
    raise FMUException(
                "Pickling is only supported for models in their initial (constructed) state. "
                "Cannot pickle a model that has been instantiated."

for my use-case (parallelizing simulations with different inputs on an uninitialized model),
the __reduce__ approach works fine - each worker gets a fresh construct, then simulates.

would you reconsider the proposal with this slightly modified approach ?

[PeterMeisrimelModelon]

I'll have to check with some of my colleagues after the summer vacations.
I have a few reservations about features that will only work correctly in very limited circumstances.

But on a high-level, your use case is that you want to run multiple (threaded) simulations of the same FMU, but only want to parse the modelDescription.xml once? Or are there other considerations I am missing?

[jschueller]

the use-case is loading the fmu, share the FMU2model instance across several worker processes, each worker simulates it from scratch with different parameters, then gather the results of the simulation, so the FMU2model instance serialization just has to work from a blank state, not from an intermediate simulation state

[jschueller]

anyway its low-risk since the proposed solution clearly raises an error if the model is already initialized,
that could be lifted later if someone decides to implement the full serialization

[jschueller]

what do you think @PeterMeisrimelModelon ?