mondras
12/06/2024, 9:41 PMTraceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/prefect/task_engine.py", line 155, in compute_transaction_key
key = self.task.cache_policy.compute_key(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/cache_policies.py", line 168, in compute_key
policy_key = policy.compute_key(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/cache_policies.py", line 168, in compute_key
policy_key = policy.compute_key(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/cache_policies.py", line 296, in compute_key
return hash_objects(hashed_inputs, raise_on_failure=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/utilities/hashing.py", line 66, in hash_objects
return stable_hash(cloudpickle.dumps((args, kwargs)), hash_algo=hash_algo)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/cloudpickle/cloudpickle.py", line 1529, in dumps
cp.dump(obj)
File "/usr/local/lib/python3.11/site-packages/cloudpickle/cloudpickle.py", line 1295, in dump
return super().dump(obj)
^^^^^^^^^^^^^^^^^
TypeError: cannot pickle 'SSLContext' object
Marvin
12/06/2024, 9:42 PMplease fix```Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/prefect/task_engine.py", line 155, in compute_transaction_key key = self.task.cache_policy.compute_key( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect/cache_policies.py", line 168, in compute_key policy_key = policy.compute_key( ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect/cache_policies.py", line 168, in compute_key policy_key = policy.compute_key( ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect/cache_policies.py", line 296, in compute_key return hash_objects(hashed_inputs, raise_on_failure=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect/utilities/hashing.py", line 66, in hash_objects return stable_hash(cloudpickle.dumps((args, kwargs)), hash_algo=hash_algo) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/cloudpickle/cloudpickle.py", line 1529, in dumps cp.dump(obj) File "/usr/local/lib/python3.11/site-packages/cloudpickle
mondras
12/06/2024, 9:42 PMTypeError: cannot pickle 'SSLContext' object
Marvin
12/06/2024, 9:42 PMSSLContext
object, often as part of tasks that involve network connections in Python. In the context of Prefect, it might happen if you're attempting to pass an object involving an SSL connection between distributed flow tasks.
Could you please confirm or specify which version of Prefect you are using? This will help me provide the most accurate solution.mondras
12/06/2024, 9:43 PMMarvin
12/06/2024, 9:43 PM'SSLContext' object
pickling issue. However, based on typical scenarios where this error might occur, it's likely due to the attempt to serialize networking objects that aren't natively serializable by Python's pickle
module.
In Prefect tasks, it's essential to avoid passing objects that need SSL contexts directly between tasks, as they can't be pickled. Instead, you might consider:
1. Refactoring the Task: If possible, refactor your task so that SSL-related operations remain within the task and not passed between tasks.
2. Using Environment Variables: Ensure your SSL configurations are set globally via environment variables, so tasks can access the necessary SSL settings without directly passing SSL context objects.
3. Use of Shared Resources: Manage connections through shared resources outside of task serialization requirements.
If you're self-hosting or using specific configurations, you might find some useful setup guidance in the Prefect 3.x self-hosting documentation.
If none of this helps, just let me know, and we can keep digging—or commiserate about the joys of network programming!