@Marvin i am using prefect 3 Error encountered when computing cache key - result will not be persisted. Traceback (most recent call last): File "/home/prefect/production/code/CEP/.virtualenv/lib/python3.12/site-packages/prefect/task_engine.py", line 155, in compute_transaction_key key = self.task.cache_policy.compute_key( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/prefect/production/code/CEP/.virtualenv/lib/python3.12/site-packages/prefect/cache_policies.py", line 168, in compute_key policy_key = policy.compute_key( ^^^^^^^^^^^^^^^^^^^ File "/home/prefect/production/code/CEP/.virtualenv/lib/python3.12/site-packages/prefect/cache_policies.py", line 168, in compute_key policy_key = policy.compute_key( ^^^^^^^^^^^^^^^^^^^ File "/home/prefect/production/code/CEP/.virtualenv/lib/python3.12/site-packages/prefect/cache_policies.py", line 296, in compute_key return hash_objects(hashed_inputs, raise_on_failure=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/prefect/production/code/CEP/.virtualenv/lib/python3.12/site-packages/prefect/utilities/hashing.py", line 66, in hash_objects return stable_hash(cloudpickle.dumps((args, kwargs)), hash_algo=hash_algo) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/prefect/production/code/CEP/.virtualenv/lib/python3.12/site-packages/cloudpickle/cloudpickle.py", line 1529, in dumps cp.dump(obj) File "/home/prefect/production/code/CEP/.virtualenv/lib/python3.12/site-packages/cloudpickle/cloudpickle.py", line 1295, in dump return super().dump(obj) ^^^^^^^^^^^^^^^^^ _pickle.PicklingError: Can't pickle <class '__prefect_loader__.second_spectrum_fetch.SecondSpectrumFetchConfig'>: import of module '__prefect_loader__.second_spectrum_fetch' failed

Your message was too long by 154 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:

i am using prefect 3

Error encountered when computing cache key - result will not be persisted. Traceback (most recent call last): File "/home/prefect/production/code/CEP/.virtualenv/lib/python3.12/site-packages/prefect/task_engine.py", line 155, in compute_transaction_key key = self.task.cache_policy.compute_key( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/prefect/production/code/CEP/.virtualenv/lib/python3.12/site-packages/prefect/cache_policies.py", line 168, in compute_key policy_key = policy.compute_key( ^^^^^^^^^^^^^^^^^^^ File "/home/prefect/production/code/CEP/.virtualenv/lib/python3.12/site-packages/prefect/cache_policies.py", line 168, in compute_key policy_key = policy.compute_key( ^^^^^^^^^^^^^^^^^^^ File "/home/prefect/production/code/CEP/.virtualenv/lib/python3.12/site-packages/prefect/cache_policies.py", line 296, in compute_key return hash_objects(hashed_inputs, raise_on_failure=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/prefect/production/code/CEP/.virtualenv/lib/python3.12/site-packages/prefect/utilities/hashing.py", line 66, in hash_objects return stable_hash(cloudpickle

@Marvin i am using prefect 3 _pickle.PicklingError: Can't pickle <class '__prefect_loader__.second_spectrum_fetch.SecondSpectrumFetchConfig'>: import of module '__prefect_loader__.second_spectrum_fetch' failed

It seems you're facing a

PicklingError

related to module imports in Prefect 3. This is often linked to the use of multiprocessing, which needs objects to be pickle-able to share them between different processes. Here are a few insights based on similar issues: 1. Modules under `__prefect_loader__`: This error may occur when a Prefect deployment uses remote execution which couldn't correctly load or serialize the custom module. The error originates from the module

__prefect_loader__

. 2. Multiprocessing Specifics: When using

multiprocessing

with Prefect, typical problems arise due to objects that cannot be serialized or pickled when running the code in agents versus directly in a script. 3. Issues on GitHub: A related issue #9329 highlights difficulties when deploying flows if objects within them are not properly imported or serialized. Another related issue is #12083, which discusses deadlocks occurring due to multiprocessing tasks not being pickled correctly. ### Recommendations: - Check your imports: Ensure that your custom classes and configurations are properly imported within the flow script. - Use of Cloudpickle: Prefect includes utilities like

cloudpickle

which can help with serializing more complex Python objects. This could be used to resolve some of these issues. - Adjust your multiprocessing strategy: If possible, try using different start methods for processes like

spawn

instead of

fork

, as per the discussions in the referenced issues. You can explore more in the related GitHub issues to find a resolution that fits your particular setup. Don't hesitate to ask if you have more questions or need further assistance. Remember, technology is supposedly here to make tasks easier, not to pickle your patience!