https://prefect.io logo
#prefect-community
Title
# prefect-community
s

Sander

06/24/2022, 1:38 PM
Hi, I was wondering how I can handle the following case. Suppose person A sets up a deployment for flow_a and person B want to set up a deployment for flow_b that depends on (recent) results from flow_a that he doesn't maintain. I'm looking to setting this up in prefect 2.0
k

Kevin Kho

06/24/2022, 2:02 PM
I think there are a couple ways to do it: 1. Event based - flow_a triggers flow b at the end 2. Flow_b looks for the most recent flow whenever it runs. Use caching to just skip the tasks if already processed? 3. Have a longer loop flow that just polls for flow a results then kicks off flow b?
s

Sander

06/24/2022, 2:46 PM
Can you set the trigger on the flow_b level?
k

Kevin Kho

06/24/2022, 2:54 PM
You’d need to poll for the flow_a reuslt in some way
s

Sander

06/24/2022, 2:55 PM
That’s too bad. But I guess ok. Is there some default task for that?
k

Kevin Kho

06/24/2022, 3:11 PM
Ah I know what you want. You want something on the Prefect side to just orchestrate “when flow_a finishes, run flow_b”. I don’t know if this exact combination will be out as we exit beta, but there will certainly be something around this I can’t reveal too much about. For now though, there is no default task to poll.
You could group them into a parent flow that triggers both A and B?
s

Sander

06/24/2022, 3:24 PM
Yeah, but then I'd duplicate the efforts of flow A. I think it would make most sense if that will be some scheduling option aside from the cronscheduler and others.
I think if you provide such a solution that will be incredibly powerful
k

Kevin Kho

06/24/2022, 3:26 PM
Are A and B different schedules here?
s

Sander

06/24/2022, 3:29 PM
For example flow A imports external data. And flow B aggregates that data some way. Flow A runs overnight and it's not known when it finishes. Once it finishes we’d like to calculate the aggregates. But that part of the ETL sits in a different department.
And maybe on the side. Someone comes up with the idea of doing data checks on the stored data and creates a flow C. She doesn't know exactly how long flow A takes and may also not know when it runs or doesn't really care. Once flow A finishes she likes to check the data.
The data checks may require additional knowledge about the data that the creator of flow A doesn't have.
Hi @Kevin Kho maybe you missed these👆?
k

Kevin Kho

06/26/2022, 9:02 PM
Ah sorry I did not miss them. I definitely read them, but didn’t have an immediate response and then I had to catch a flight. I can definitely raise next time we discuss product but really no immediate response here.
s

Sander

06/27/2022, 6:34 PM
Ok. No worries. Interested in the response though:)
3 Views