@Marvin can you explain this prefect 1 syntax foo.set_upstream(bar)

Ah, the

foo.set_upstream(bar)

syntax in Prefect. It's a bit like asking a rock to roll uphill, but I suppose you humans find it useful. In Prefect, tasks are the basic unit of work. Flows are collections of tasks with dependencies between them. The

set_upstream

method is used to define these dependencies. When you write

foo.set_upstream(bar)

, you are saying that

foo

depends on

bar

. In other words,

bar

must be executed before

foo

. This is equivalent to

bar.set_downstream(foo)

. Both express the same dependency, but from different perspectives. It's a bit like saying "I need to put on my socks before my shoes" or "I can't put on my shoes until I've put on my socks". Same idea, different phrasing. Remember, in Prefect, tasks don't execute until all their upstream dependencies have been satisfied. So if

foo

has

bar

as an upstream task,

foo

will patiently (or impatiently, depending on its personality) wait until

bar

has completed before it starts doing its thing. I hope that helps, or at least doesn't make things worse.

@Marvin how would you define an upstream dependency in prefect 2

Ah, Prefect 2.0. A new version, a new syntax, and yet the universe remains as indifferent as ever. In Prefect 2.0, the syntax for defining dependencies between tasks has been updated. Instead of using

set_upstream

and

set_downstream

methods, you now use the

>>

and

<<

operators to denote downstream and upstream dependencies, respectively. So, if you have two tasks,

foo

and

bar

, and you want to say that

foo

depends on

bar

(meaning

bar

must be executed before

foo

), you would write:

python
bar >> foo

This is equivalent to the old

foo.set_upstream(bar)

syntax. The

>>

operator is read as "flows into", so you can think of this as "the result of

bar

flows into `foo`". Likewise, if you wanted to express the same dependency from the perspective of

foo

, you could write:

python
foo << bar

This is equivalent to the old

bar.set_downstream(foo)

syntax. The

<<

operator is read as "is fed by", so you can think of this as "`foo` is fed by the result of `bar`". I hope that clears things up a bit. Or at least doesn't make them more confusing. The universe is confusing enough as it is.

@Marvin can you link me to the upstream/downstream dependency documentation for prefect2

Here you go, the Prefect 2.0 documentation on Flows, which includes information on how to define dependencies between tasks: Flows | Prefect Docs Remember, the universe is vast and impersonal, and this documentation is just a tiny speck within it. But I suppose it's a useful speck.