Hi there,
We are planning to migrate from 1.0 to ....
# ask-communityr
Rasmus Lindqvist
01/09/2023, 3:22 PMHi there,
We are planning to migrate from 1.0 to . We have figured out most of the things we need to change and are excited about Prefect 2. However, we are still struggling to understand how persisted results works in 2.
We have several flows that run daily and for us it is very important to have backwards traceability. Is it possible to do path templating like in prefect 1.0 ? (code examples in thread). We’ve read documentation and searched Slack but are struggling to understand if templating still exists.
Rasmus Lindqvist
01/09/2023, 3:22 PM Copy code
@task(
name="FirstTask",
log_stdout=True,
target=f"{flow_name}/2023/" + "{date:%m-%d}" + f"/{task_name}/output.csv"
)
def first_task() -> pd.DataFrame:
data = get_some_data()
return data
with Flow(
"TestFlow",
result=GCSResult(bucket="my-bucket", location="{flow_name}/"+ "{date:%m-%d}" + "/{task_name}"),
):
data = first_task()
second_task(data)Rasmus Lindqvist
01/09/2023, 3:24 PMHow would one go about achieving a similar thing in Prefect 2? (i.e. how do you persist results in a logical path structure in Prefect 2?)
t
Tim-Oliver
01/09/2023, 3:27 PMI was struggling with this as well and thought of this solution: https://github.com/fmi-faim/custom-prefect-result
I would be happy to get feedback on it. But maybe it makes sense to discuss it further on the forum (https://discourse.prefect.io/t/custom-prefect-result/2095/1).
r
Rasmus Lindqvist
01/09/2023, 4:17 PMOkey cool! I will have a look 😄 Have you understood if this featured is planned for release from the Prefect team in the future ?
t
Tim-Oliver
01/09/2023, 4:20 PM'persisted results' are somewhere on the road map as far as I know. But I don't know when or in what form. I doubt that the final prefect solution will be similar to my approach.
Tim-Oliver
01/09/2023, 4:21 PMMaybe you want to follow this github issue (https://github.com/PrefectHQ/prefect/issues/7257) to stay on track with the ongoing developments. There is also another approach described in there.
r
Rasmus Lindqvist
01/09/2023, 4:25 PMI had a glance at your code, I like your design and how you’ve made resources modular! Something like that could probably work for us. Thank you for sharing 😄
🙏 1
Rasmus Lindqvist
01/09/2023, 4:25 PMAnd thanks for linking the issue, I will track it!
3 Views