Hello, Is there a way to export Prefect flow defin...
# ask-community
s
Hello, Is there a way to export Prefect flow definition to some config format like YAML, and then re-build the flow based on the same imported config?
z
Hi @Satheesh K -- there's actually a new feature that does this. https://github.com/PrefectHQ/prefect/pull/4317
Let me see if I can find where a better example is...
🙏 1
Ah yeah you'll just want to use
prefect build --help
s
Hi Micheal, Thanks for your response. Let me check. For reference, we want to do something like this https://donagh.io/configuring-airflow-dags-with-yaml/
d
Hi @Satheesh K! I’m curious: what sort of configuration would you want to configure it YAML that the Prefect Cloud/Server UI doesn’t cover? Are you migrating directly from Airflow? Is there a specific feature you’re looking for a 1:1 match for?
Put another way: can you outline the business / overall goal of configuring Flows this way? There might be a more “prefect” way 😄
s
Okay, sorry if I sound dumb, I have recently discovered prefect, am not fully knowledgeable about Cloud UI. Our use-case is running multiple ML models and do some post-processing and consolidate at the end. something like below:
Copy code
flow:
	task:
		name: run_detection_model
		inputs: [model1,model2,model3]
		output: [model1_dets,model2_dets,model3_det]

	task:
		name: do_something_with_model1_dets
		inputs: [model1_dets]
		output: [model1_dets_filtered]

....
Here task->name is a perfect task
these tasks are general enough that they will also be used in other flows as well.
d
You’ve come to the right place to ask questions!
I would recommend two approaches depending on which is a better fit for you
First, if you’re looking to create a common library of Tasks that will be used by many Flows (things like making SQL queries against a particular database at your organization), I would recommend subclassing
Task
and creating your own specialized Prefect Tasks. Then, package them up and share that code in a python package. Take a look at our own Task Library for some inspiration: https://docs.prefect.io/core/task_library/overview.html
If you have a few “Sub-Flows” or sections of a Flow that are common logic and may be used in other Flows, I would recommend looking a Flows of Flows (also known as the Orchestrator Pattern): https://docs.prefect.io/core/idioms/flow-to-flow.html
Would either of these patterns work for what you’re hoping to achieve?
s
We are already kind of following the first approach, keep all the common tasks in one place by subclassing Prefect Tasks. But this will lead us one
.py
file for each flow. I think we are looking more at something like having one file
decode_flow.py
to decode DAG definition written in yaml config etc. Here I have to mention all our tasks and flows are very much similar, but we also thinking about extensibility in future.
Thanks for the
Sub-Flows
will check it in more detail.
d
Anytime!
In case you haven’t seen them yet, Parameters may also be useful: https://docs.prefect.io/core/concepts/parameters.html
👍 1