Andreas
05/30/2022, 3:16 PMflow.add_edge(_upstream_task_, _downstream_task_, _*key*_)
is there any way to specify which of the multiple return values from the upstream task should pass to the downstream task?Anna Geller
Kevin Kho
Andreas
05/31/2022, 9:47 AMAnna Geller
by using the API to create the flowsyou can think of Python as an API for Prefect 2.0 - instead of having a hard-coded set of dependencies defined at build time, you can have a dynamically defined computational graph by doing nothing more than updating your code
Andreas
06/01/2022, 11:53 AMflow.add_edge
. In this way I avoid manually writing the code to define each flow. I want to transition to Prefect 2.0 but if there isn't a secondary way to set those dependencies between tasks like in Prefect 1.0 this seems really hardAnna Geller
avoid manually writing the code to define each flowit may be an oversimplification from my side, but I think: you either write Python code or you write a JSON config + Python file consuming that JSON file Using Python without JSON DAG definition makes your architecture a bit simpler since there is only one place to adjust the changes and it seems a bit friendlier? it's a matter of personal preference you can certainly read JSON in your flow code and start there I think the underlying problem you were trying to solve with JSON config is reducing boilerplate - would you agree?
Andreas
06/01/2022, 1:42 PMAnna Geller
Andreas
06/01/2022, 2:10 PMflow.add_edge(_upstream_task_, _downstream_task_, _*key*_)
works fine but only for single return parameters under the key
attribute
2. Is there an alternative way to create dependencies in Prefect 2.0 (or planned) apart from (the task parameter `wait_for` ) like it is easily done in Prefect 1.0 with add_edge?Anna Geller
I told you about it because you asked about a use case :)gotcha, thanks for that #1 not sure how to do it imperatively, sorry #2 again here, I won't have any encouraging answer because this entire imperative API was based on the concept of built-time DAG which Prefect 2.0 elevates to a runtime-DAG - that's why dependencies are not defined by DAG nodes and edges, but via passing data between tasks + state dependencies via
wait_for
- there is certainly a solution for your problem in Prefect 2.0, but using plain Python rather than class-based DAG API, e.g. you can add tasks and their dependencies dynamically based on what you read from those JSON files - this could be the first step before you migrate it to be Python codeAndreas
06/01/2022, 3:13 PM