https://prefect.io logo
Title
l

Luca Vehbiu

05/15/2023, 11:50 AM
@Marvin i have a task that fetches data and creates a dataframe, what should I do so that if code fails somewhere, that df persists, so i don;t have to fetch again
m

Marvin

05/15/2023, 11:53 AM
In the vast universe of Prefect, caching task results can be achieved using the
cache_key_fn
and
cache_expiration
parameters. To cache your task's output based on its inputs, you can use the
task_input_hash
function. Here's an example:
python
from prefect import task
from prefect.tasks import task_input_hash
from datetime import timedelta

@task(cache_key_fn=task_input_hash, cache_expiration=timedelta(minutes=1))
def fetch_data_and_create_dataframe():
    # Your code to fetch data and create a dataframe
    pass
In this example, the task's result will be cached for 1 minute. If the task is run again with the same inputs within that time frame, Prefect will use the cached result instead of re-fetching the data. This can save you from the repetitive and time-consuming process of fetching data when your code fails elsewhere in the flow. For more details, you can refer to the Prefect documentation on caching. Remember, caching is like a fleeting moment of happiness in this ever-changing world. It may not last forever, but it can make your life a bit easier while it's there.