Ville Parkkinen

09/28/2022, 1:00 PM
Hi, I’m having trouble using cache with tasks that take blocks as parameters. It seems that the
hashes the block differently on each flow run, and therefore causing cache misses. I’ll attach code snippets needed to reproduce the issue in this thread.
Define and save a custom block for API url/credentials:
# blocks/

from pydantic import SecretStr

from prefect.blocks.core import Block

class ApiBlock(Block):
    base_url: str
    path: str
    api_key: SecretStr

test_block = ApiBlock(base_url="<|>", path="data", api_key="secret_key")"test-block", overwrite=True)
Flow definition with two almost identical tasks, one takes a block as parameters and the other takes strings:

from datetime import timedelta
from pydantic import SecretStr

from prefect import flow, task, get_run_logger
from prefect.tasks import task_input_hash

from blocks.api_block import ApiBlock

@task(cache_key_fn=task_input_hash, cache_expiration=timedelta(minutes=10))
def call_api_with_block(api_conf: ApiBlock):
    logger = get_run_logger()
    <|>(f"Getting data from {api_conf.base_url}/{api_conf.path}, with key: {api_conf.api_key}")
    return "some data...."

@task(cache_key_fn=task_input_hash, cache_expiration=timedelta(minutes=10))
def call_api_with_strings(base_url: str, path: str, api_key: SecretStr):
    logger = get_run_logger()
    <|>(f"Getting data from {base_url}/{path}, with key: {api_key}")
    return "some data...."

def get_data_from_api():
    test_block = ApiBlock.load("test-block")

    call_api_with_strings(test_block.base_url, test_block.path, test_block.api_key)

Running the
flow multiple times shows that
is executed with every run, while
is cached correctly. Wrapping the
with a custom function and logging the cache key shows that the key is different on every run:
def cache_key_logger(context, params):
    logger = get_run_logger()
    cache_key = task_input_hash(context, params)
    return cache_key
I’m using Prefect 2.4.2

Khuyen Tran

09/28/2022, 3:31 PM
Thanks for raising this issue. This seems to be a bug. Do you mind opening the issue on Prefect GitHub so we can take a closer look at this?

Ville Parkkinen

09/29/2022, 4:42 AM
Sure, I’ll write up an issue!