https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> i have just updated prefect from 2.20...
# ask-marvin
a
@Marvin i have just updated prefect from 2.20.16 to 3.6.6. I am getting this error when i run flows:
Copy code
04:42:34.076 | ERROR   | Task run 'extract-cb6' - Error encountered when computing cache key - result will not be persisted.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/cache_policies.py", line 374, in compute_key
    return hash_objects(hashed_inputs, raise_on_failure=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/utilities/hashing.py", line 89, in hash_objects
    raise HashError(msg)
prefect.exceptions.HashError: Unable to create hash - objects could not be serialized.
  JSON error: Unable to serialize unknown type: <class 'sqlalchemy.engine.base.Engine'>
  Pickle error: cannot pickle '_thread.RLock' object

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/task_engine.py", line 282, in compute_transaction_key
    key = self.task.cache_policy.compute_key(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/cache_policies.py", line 214, in compute_key
    policy_key = policy.compute_key(
                 ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/cache_policies.py", line 384, in compute_key
    raise ValueError(msg) from exc
ValueError: Unable to create hash - objects could not be serialized.
  JSON error: Unable to serialize unknown type: <class 'sqlalchemy.engine.base.Engine'>
  Pickle error: cannot pickle '_thread.RLock' object

This often occurs when task inputs contain objects that cannot be cached like locks, file handles, or other system resources.

To resolve this, you can:
  1. Exclude these arguments by defining a custom `cache_key_fn`
  2. Disable caching by passing `cache_policy=NO_CACHE`
What does mean and how do i address it?
m
✅ thought for 178.5 seconds
Thanks for sharing the traceback — this is a common change you’ll hit going from Prefect 2.x to 3.x. What’s happening - In Prefect 3.x, tasks compute a cache/transaction key by default using a policy that includes task inputs. - One of your task’s inputs is a SQLAlchemy 
Engine
, which contains locks and other objects that cannot be serialized/hashed. - When Prefect tries to hash that input, it fails and logs “Unable to create hash… result will not be persisted.” The task may still run, but it won’t be able to use caching for that run. How to fix it Pick one of these approaches: 1) Exclude the unhashable parameter from the cache key (recommended) - If your task has a parameter like 
engine
, exclude it from the default policy:
Copy code
from prefect import task
from prefect.cache_policies import DEFAULT

@task(cache_policy=DEFAULT - "engine", persist_result=True)
def extract(engine, query):
    # your code
    ...
- Replace 
"engine"
with whatever the arg name is (e.g., 
"session"
, 
"conn"
). 2) Provide a custom cache_key_fn (full control) - Build the key from only the parameters that matter and are hashable:
Copy code
from prefect import task

def cache_on_query_only(context, params):
    # e.g., only cache based on the query text
    return f"extract:{params.get('query','')}"

@task(cache_key_fn=cache_on_query_only, persist_result=True)
def extract(engine, query):
    # your code
    ...
Notes: - Signature is 
cache_key_fn(context, parameters) -> Optional[str]
. - Return 
None
to skip caching for a given call. 3) Disable caching for that task - If you don’t need caching for this task:
Copy code
from prefect import task
from prefect.cache_policies import NO_CACHE

@task(cache_policy=NO_CACHE)
def extract(engine, query):
    ...
- Alternatively, 
@task(persist_result=False)
will also result in no caching. Optional global switches (if you want old 2.x behavior) - Disable all task caching: - 
PREFECT_TASKS_DISABLE_CACHING=true
- Make no-cache the default for new tasks: - 
PREFECT_TASKS_DEFAULT_NO_CACHE=true
If you share your task signature (which arg is the 
Engine
), I can suggest the exact decorator change for your code.
5 Views
Open in Slack
PreviousNext
Powered by