Philip Blankenau
04/22/2020, 11:42 PMJeremiah
04/22/2020, 11:47 PMpath
as the task’s output; then the downstream task accepts path
as an argument and uses it to load the data.
The important thing is that even though Prefect has facilities for exchanging data between tasks, it is agnostic to what that data is (or if you pass any data at all!)
In addition, the next Prefect release will include a refactored Result
class that will make some of the details of serializing / deserializing data to disk easier (whether local, cloud, or anywhere)Philip Blankenau
04/22/2020, 11:59 PMJeremiah
04/23/2020, 12:02 AMsubprocess.check_call()
subprocess doesn’t raise an error on failure. Prefect will trap the error and interpret the task as failing; if the task function runs to completion Prefect treats it as a success.Philip Blankenau
04/23/2020, 2:57 PMJeremiah
04/23/2020, 2:58 PMtarget
, similar to Luigi) in order to help users who do want artifact-based states.Philip Blankenau
04/23/2020, 3:35 PMJeremiah
04/23/2020, 3:37 PMPhilip Blankenau
04/23/2020, 9:23 PMJeremiah
04/23/2020, 10:49 PMflow.run()
and let it run.