Chris Hart
07/02/2019, 6:19 PMjosh
07/02/2019, 6:51 PMChris Hart
07/02/2019, 6:54 PMjosh
07/02/2019, 7:00 PMprojects
which can contain as many flows (and as many versions of those flows) as you would like. Projects in this sense act as a folder structure.Chris Hart
07/02/2019, 7:09 PMDavid Ojeda
07/03/2019, 10:28 AMxxx
functions
tasks
flows
The xxx
module is our “data pipeline package”.
The xxx.functions
module has the low-level functions or scripts that do most of the important work. These are written without any prefect knowledge. An example function: bandpass filter a file with some signals.
The xxx.tasks
has objects that derive from prefect.Task
and manage the inputs, outputs, parameters of one or many functions in the xxx.functions
module. The usage of these functions gives some semantic meaning to the task. So far for us, the line between task and function is still blurry, but sometimes it makes sense to group several functions together to avoid sending a lot of data between workers.
The xxx.flows
then orchestrates several tasks defined in xxx.tasks
. It connects them and manages how input/outputs are connected, how failures are managed, logging, etc.josh
07/03/2019, 12:40 PMChris White
Marvin
08/05/2019, 9:23 PM