David Kuda

    David Kuda

    1 year ago
    Hello everyone, we have been testing Prefect since December, and so far we really like it and are excited to test it on live projects. I am by now quite familiar with the docs and with orchestration, and I would like to ask for some help on one thing which is not entirely clear yet to me. I want to give an example: Let’s say we have a flow that does SQL stuff on Postgres, for example
    1) create table a
    ,
    2) create table b
    and
    3) join data from tables a and b
    . We want to use the dask executor, because there are numerous more tasks. What is the recommended way to have
    task 1
    and
    task 2
    run simultaneously, but have
    task 3
    wait for tasks 1 and 2? The dask executor would create a DAG where it would run tasks 1, 2 and 3 together, but that will fail, since task 3 is depending on tasks 1 and 2. The Python code however, has no dependency. I have few ideas, and I am excited to see your opinion on that question. I hope that I could express myself clearly. Best regards from Berlin, David.
    j

    josh

    1 year ago
    Hi @David Kuda you can set task upstreams and downstreams directly through the API with something like this:
    In [1]: with Flow("my_flow") as flow:
       ...:     t1 = my_task1()
       ...:     t2 = my_task2()
       ...:     t3 = my_task3()
       ...:
       ...:     t3.set_upstream(t1)
       ...:     t3.set_upstream(t2)
    This will produce a dag that looks like this where tasks 1 and 2 can execute in parallel and task 3 won’t execute until they finish:
    David Kuda

    David Kuda

    1 year ago
    Hey @josh — Thank you so much for your fast and helpful reply, that’s great! Do you have a link to the docs that gives further details on that?
    I am excited to try this out! 🙂
    Michael Adkins

    Michael Adkins

    1 year ago
    btw, you guys have an amazing docs page, and code documentation, too. Thank you for offering that!
    merlin

    merlin

    1 year ago
    Yep the documentation and clean look to the code is what has attracted me to Prefect.