https://prefect.io logo
Powered by
Title
r

Rasmus Lindqvist

01/09/2023, 3:22 PM
Hi there, We are planning to migrate from 1.0 to . We have figured out most of the things we need to change and are excited about Prefect 2. However, we are still struggling to understand how persisted results works in 2. We have several flows that run daily and for us it is very important to have backwards traceability. Is it possible to do path templating like in prefect 1.0 ? (code examples in thread). We’ve read documentation and searched Slack but are struggling to understand if templating still exists.
@task(
    name="FirstTask",
    log_stdout=True,
    target=f"{flow_name}/2023/" + "{date:%m-%d}" + f"/{task_name}/output.csv"
)
def first_task() -> pd.DataFrame:
    data = get_some_data()
    return data

with Flow(
    "TestFlow",
    result=GCSResult(bucket="my-bucket", location="{flow_name}/"+ "{date:%m-%d}" + "/{task_name}"),
):
    data = first_task()
    second_task(data)
How would one go about achieving a similar thing in Prefect 2? (i.e. how do you persist results in a logical path structure in Prefect 2?)
t

Tim-Oliver

01/09/2023, 3:27 PM
I was struggling with this as well and thought of this solution: https://github.com/fmi-faim/custom-prefect-result I would be happy to get feedback on it. But maybe it makes sense to discuss it further on the forum (https://discourse.prefect.io/t/custom-prefect-result/2095/1).
r

Rasmus Lindqvist

01/09/2023, 4:17 PM
Okey cool! I will have a look 😄 Have you understood if this featured is planned for release from the Prefect team in the future ?
t

Tim-Oliver

01/09/2023, 4:20 PM
'persisted results' are somewhere on the road map as far as I know. But I don't know when or in what form. I doubt that the final prefect solution will be similar to my approach.
Maybe you want to follow this github issue (https://github.com/PrefectHQ/prefect/issues/7257) to stay on track with the ongoing developments. There is also another approach described in there.
r

Rasmus Lindqvist

01/09/2023, 4:25 PM
I had a glance at your code, I like your design and how you’ve made resources modular! Something like that could probably work for us. Thank you for sharing 😄
🙏 1
And thanks for linking the issue, I will track it!
3 Views
#prefect-communityJoin Slack
Powered by