https://prefect.io
Join Slack
Hi there, We are planning to migrate from 1.0 to . We have figured out most of the things we need to...
# ask-community
r
Hi there, We are planning to migrate from 1.0 to . We have figured out most of the things we need to change and are excited about Prefect 2. However, we are still struggling to understand how persisted results works in 2. We have several flows that run daily and for us it is very important to have backwards traceability. Is it possible to do path templating like in prefect 1.0 ? (code examples in thread). We’ve read documentation and searched Slack but are struggling to understand if templating still exists.
Copy code
@task(
    name="FirstTask",
    log_stdout=True,
    target=f"{flow_name}/2023/" + "{date:%m-%d}" + f"/{task_name}/output.csv"
)
def first_task() -> pd.DataFrame:
    data = get_some_data()
    return data

with Flow(
    "TestFlow",
    result=GCSResult(bucket="my-bucket", location="{flow_name}/"+ "{date:%m-%d}" + "/{task_name}"),
):
    data = first_task()
    second_task(data)
How would one go about achieving a similar thing in Prefect 2? (i.e. how do you persist results in a logical path structure in Prefect 2?)
t
I was struggling with this as well and thought of this solution: https://github.com/fmi-faim/custom-prefect-result I would be happy to get feedback on it. But maybe it makes sense to discuss it further on the forum (https://discourse.prefect.io/t/custom-prefect-result/2095/1).
r
Okey cool! I will have a look 😄 Have you understood if this featured is planned for release from the Prefect team in the future ?
t
'persisted results' are somewhere on the road map as far as I know. But I don't know when or in what form. I doubt that the final prefect solution will be similar to my approach.
Maybe you want to follow this github issue (https://github.com/PrefectHQ/prefect/issues/7257) to stay on track with the ongoing developments. There is also another approach described in there.
r
I had a glance at your code, I like your design and how you’ve made resources modular! Something like that could probably work for us. Thank you for sharing 😄
🙏 1
And thanks for linking the issue, I will track it!
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by