https://prefect.io logo
Title
q

Qwame

08/09/2020, 9:06 AM
Hello everyone, I have a task that several other tasks depend on in my prefect flow. In airflow I could do something like
F1 >> [f2, f3, f4, f5, f6]
What's the best way to set these dependencies in Prefect. I notice that set_downstream doesn't accept a list of tasks. Is there any efficient way to do this in Prefect? Also does the new Prefect UI mean I don't need docker to run it? Thanks
n

nicholas

08/09/2020, 11:25 AM
Hi @Qwame - you can set upstream or downstream tasks by passing an array to the
upstream_tasks
or
downstream_tasks
task constructor, or using the
set_dependencies
method like such:
from prefect import Flow, task
 
@task
def F1():
    print("F1")
    return

@task
def f2():
    print("f2")

@task
def f3():
    print("f3")

with Flow("F Flow") as flow:
    f1 = F1()

    f2(upstream_tasks=[f1])
    f3(upstream_tasks=[f1])
or:
from prefect import task, Flow

@task
def F1():
    print("F1")

@task
def f2():
    print("f2")

@task
def f3():
    print("f3")
    
with Flow("F Flow") as flow:
    F1.set_dependencies(downstream_tasks=[f2, f3])
As for Prefect Server (and Prefect UI), each of the repos can be run without Docker, however Docker (and docker-compose) are the container/service orchestration tools we've chosen to make it simple to spin everything up at once; without Docker, right now you'll need to install dependencies and run each service individually.
q

Qwame

08/10/2020, 1:34 AM
Thank you very much @nicholas
n

nicholas

08/10/2020, 1:37 AM
Happy to help!
q

Qwame

08/10/2020, 1:44 AM
It is interesting how the second example is set up
Usually I have to do something like F1 = F1() `f`2 = f2() f3 = f3() Before we can do this:
F1.set_dependencies(downstream_tasks=[f2, f3])
I didn't know we could call the set_dependencies without doing the above first.
n

nicholas

08/10/2020, 2:58 AM
Both are valid; one builds the dependency graph based on the reference to the task call (which is your example), and the other dependency graph is built based on Prefect's knowledge of your intention to call a task (understood from your passing a reference to the task itself)
👍 1
q

Qwame

08/10/2020, 6:24 AM
I have tried set_dependencies using both the functional and imperative API and I must admit that I'm very happy with Prefect. I haven't been successful at using the UI without docker. I am unable to use docker that's why I have to install and run each service individually. I cloned the repo, and followed the instructions to build the Prefect development server. The UI showed in the browser at port 8080 but I couldn't connect my flows to the UI. I used flow.register() but it didn't work. The error was related to connection to the port. Any idea what I might be missing?
a

Archie35

10/06/2020, 10:36 PM
@Qwame did you ever figure out what was wrong? I think I am having a similar connection issue
q

Qwame

10/07/2020, 12:24 AM
@Archie35 nope I could not figure out what was wrong.