https://prefect.io logo
Title
d

David Kuda

01/27/2021, 9:46 PM
Hello everyone, we have been testing Prefect since December, and so far we really like it and are excited to test it on live projects. I am by now quite familiar with the docs and with orchestration, and I would like to ask for some help on one thing which is not entirely clear yet to me. I want to give an example: Let’s say we have a flow that does SQL stuff on Postgres, for example
1) create table a
,
2) create table b
and
3) join data from tables a and b
. We want to use the dask executor, because there are numerous more tasks. What is the recommended way to have
task 1
and
task 2
run simultaneously, but have
task 3
wait for tasks 1 and 2? The dask executor would create a DAG where it would run tasks 1, 2 and 3 together, but that will fail, since task 3 is depending on tasks 1 and 2. The Python code however, has no dependency. I have few ideas, and I am excited to see your opinion on that question. I hope that I could express myself clearly. Best regards from Berlin, David.
j

josh

01/27/2021, 9:49 PM
Hi @David Kuda you can set task upstreams and downstreams directly through the API with something like this:
In [1]: with Flow("my_flow") as flow:
   ...:     t1 = my_task1()
   ...:     t2 = my_task2()
   ...:     t3 = my_task3()
   ...:
   ...:     t3.set_upstream(t1)
   ...:     t3.set_upstream(t2)
:upvote: 2
This will produce a dag that looks like this where tasks 1 and 2 can execute in parallel and task 3 won’t execute until they finish:
:upvote: 2
d

David Kuda

01/27/2021, 9:51 PM
Hey @josh — Thank you so much for your fast and helpful reply, that’s great! Do you have a link to the docs that gives further details on that?
I am excited to try this out! 🙂
z

Zanie

01/27/2021, 9:56 PM
Hi @David Kuda — there’s an example at https://docs.prefect.io/core/getting_started/first-steps.html#imperative-api
:upvote: 2
btw, you guys have an amazing docs page, and code documentation, too. Thank you for offering that!
:marvin: 3
🚀 3
m

merlin

01/28/2021, 1:36 AM
Yep the documentation and clean look to the code is what has attracted me to Prefect.