Chris Hart07/02/2019, 6:19 PM
josh07/02/2019, 6:51 PM
Chris Hart07/02/2019, 6:54 PM
josh07/02/2019, 7:00 PM
which can contain as many flows (and as many versions of those flows) as you would like. Projects in this sense act as a folder structure.
Chris Hart07/02/2019, 7:09 PM
David Ojeda07/03/2019, 10:28 AM
xxx functions tasks flows
module is our “data pipeline package”. The
module has the low-level functions or scripts that do most of the important work. These are written without any prefect knowledge. An example function: bandpass filter a file with some signals. The
has objects that derive from
and manage the inputs, outputs, parameters of one or many functions in the
module. The usage of these functions gives some semantic meaning to the task. So far for us, the line between task and function is still blurry, but sometimes it makes sense to group several functions together to avoid sending a lot of data between workers. The
then orchestrates several tasks defined in
. It connects them and manages how input/outputs are connected, how failures are managed, logging, etc.
josh07/03/2019, 12:40 PM