Hello, turning again to the amazing Prefect community for help with the error below. I figured out the file that is creating the issue is my log file (which is in use during execution)... Is there a way that I can exclude it from being pickled ?

Hey @Alain Prasquier, so by default Prefect will pickle the Flow during registration, but you can avoid this by using script based storage where it will just keep the location of your Flow. If you get the chance, could you move the traceback so the thread so we can keep the main channel a bit cleaner? Also take note that if you use a DaskExecutor, stuff is sent to the workers using cloudpickle. But I guess you aren’t because the log file may not work across workers.

Thanks ! will try this ASAP.

Traceback (most recent call last):
  File "/Users/alp/Documents/GitRepo/SIPP/sipp/scenarii/enhance_images.py", line 48, in <module>
    flow.register(project_name="sipp-stage")
  File "/Users/alp/Library/Caches/pypoetry/virtualenvs/sipp-2OIxreY8-py3.9/lib/python3.9/site-packages/prefect/core/flow.py", line 1708, in register
    registered_flow = client.register(
  File "/Users/alp/Library/Caches/pypoetry/virtualenvs/sipp-2OIxreY8-py3.9/lib/python3.9/site-packages/prefect/client/client.py", line 797, in register
    serialized_flow = flow.serialize(build=build)  # type: Any
  File "/Users/alp/Library/Caches/pypoetry/virtualenvs/sipp-2OIxreY8-py3.9/lib/python3.9/site-packages/prefect/core/flow.py", line 1478, in serialize
    self.storage.add_flow(self)
  File "/Users/alp/Library/Caches/pypoetry/virtualenvs/sipp-2OIxreY8-py3.9/lib/python3.9/site-packages/prefect/storage/local.py", line 143, in add_flow
    f.write(flow_to_bytes_pickle(flow))
  File "/Users/alp/Library/Caches/pypoetry/virtualenvs/sipp-2OIxreY8-py3.9/lib/python3.9/site-packages/prefect/utilities/storage.py", line 175, in flow_to_bytes_pickle
    cloudpickle.dumps(flow, protocol=4), newline=False
  File "/Users/alp/Library/Caches/pypoetry/virtualenvs/sipp-2OIxreY8-py3.9/lib/python3.9/site-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "/Users/alp/Library/Caches/pypoetry/virtualenvs/sipp-2OIxreY8-py3.9/lib/python3.9/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
  File "/Users/alp/Library/Caches/pypoetry/virtualenvs/sipp-2OIxreY8-py3.9/lib/python3.9/site-packages/cloudpickle/cloudpickle_fast.py", line 303, in _file_reduce
    raise pickle.PicklingError(
_pickle.PicklingError: Cannot pickle files that are not opened for reading: a

Thanks for moving!

I tried changing the flow to S3 storage, but I get the exact same error

with Flow("enhance_flow", storage=S3(bucket=config.AWS_SIPP_BUCKET)) as flow:
    files = getImages()
    enhanced = enhanceImages.map(files)

S3 is still pickle by default. I think you want something like

S3(stored_as_script=True, file_path="...", bucket="...")

and this will upload your script file to S3

thank you !