Hey, I just finished going through the Get Started section for the Prefect 2.0 docs. I haven't yet g...

YSF

09/01/2022, 9:09 PM

Hey, I just finished going through the Get Started section for the Prefect 2.0 docs. I haven't yet gotten to the Concepts yet, so bear with me if this is a dumb question. But how do I define dependencies between tasks? What is the equivalent of

task.set_upstream

from prefect 1? I know Prefect 2.0 tries to remove the strict dependency on DAGs but are DAGs gone altogether?

✅ 1

YSF

09/02/2022, 2:10 AM

Update: I've finished going through all the concepts too now. I don't know if its just the documentation but it seems like actual DAGs have become somewhat relegated in terms of features and support? I really had to dig to find an example of how to set task dependencies and came across this. It says "between tasks that do not exchange data". Was that not one of the huge problems of Airflow and xcom that Prefect originally set out to solve? Its a bit ominous that this link from google returns a 404:

Taylor Curran

09/02/2022, 1:26 PM

Hi Yusuf! Prefect 2.0 does not require DAG definition: If tasks pass data to one and other, Prefect will derive their dependence on each other at runtime, that is why we provide ways for you to specify dependence if no data is passed between. If data is passed between them, no need to worry about defining the dependency structure (which may or may not be a DAG) Prefect should figure this out during runtime assuming .submit() syntax is employed.

YSF

09/02/2022, 3:32 PM

Understood, thanks that was throwing me off. I didn't gather that it was implicit

👍 1

gratitude thank you 1

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.