Hi,. I am still pretty confused. I am new to the prefect. I need to have run an ETL task. Such that Extract is done by one agent and transform /load is done by another agent .
How can I achieve that. How can I run two agents. I know I can do labels with agents. But can you refer to any example. It would help to understand better, Please guide.
k
Kevin Kho
04/29/2021, 1:37 PM
Hey @Talha! The approach here would be to attach labels to the RunConfig. The RunConfig is applied at the flow level, meaning that you need to make 1 flow for Extract and another flow for Transform/Load. These can then be tied together with another flow that calls them with the
StartFlowRun
task.
t
Talha
04/29/2021, 1:39 PM
RunConfig will help me to set the labels for the flows, but how can i run the flows on different agents?
Talha
04/29/2021, 1:39 PM
I tried running two different local agents with different labels
k
Kevin Kho
04/29/2021, 1:40 PM
Agents will only pick up flows with the same label so you provide the corresponding labels to the
Extract
flow and the
Transform/Load
flow when you define those flows.
upvote 2
Kevin Kho
04/29/2021, 1:41 PM
Use them as subflows of an outer flow and their labels will still be applied.
t
Talha
04/30/2021, 9:19 AM
Hi,
I am unable to start the prefect server
Talha
04/30/2021, 9:19 AM
The UI is not working It was working fine till yesterday. Today it has been malfunctioning
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.