https://prefect.io logo
c

Chris Hart

07/02/2019, 6:19 PM
hey peeps, any preferences for organizing projects/namespaces based on prefect flows?
j

josh

07/02/2019, 6:51 PM
What exactly do you mean by projects/namespaces?
c

Chris Hart

07/02/2019, 6:54 PM
oh just like a generic way of organizing files
(mentally comparing with Airflow's opinionated thing)
starting with a single script for now, just curious if larger production projects end up exhibiting a common pattern
j

josh

07/02/2019, 7:00 PM
Oh yeah great question! Locally flows are simply importable objects so you can really organize them any way you would like. Once flows are serialized and deployed to cloud you are able to organize them into
projects
which can contain as many flows (and as many versions of those flows) as you would like. Projects in this sense act as a folder structure.
⬆️ 1
c

Chris Hart

07/02/2019, 7:09 PM
ok cool thx!
d

David Ojeda

07/03/2019, 10:28 AM
Here is how we are organizing ourselves: We defined three submodules as follows:
Copy code
xxx
    functions
    tasks
    flows
The
xxx
module is our “data pipeline package”. The
xxx.functions
module has the low-level functions or scripts that do most of the important work. These are written without any prefect knowledge. An example function: bandpass filter a file with some signals. The
xxx.tasks
has objects that derive from
prefect.Task
and manage the inputs, outputs, parameters of one or many functions in the
xxx.functions
module. The usage of these functions gives some semantic meaning to the task. So far for us, the line between task and function is still blurry, but sometimes it makes sense to group several functions together to avoid sending a lot of data between workers. The
xxx.flows
then orchestrates several tasks defined in
xxx.tasks
. It connects them and manages how input/outputs are connected, how failures are managed, logging, etc.
j

josh

07/03/2019, 12:40 PM
@David Ojeda Yeah in a way tasks and functions can be quite similar. The best way to think about it is functions do something while tasks are Prefect’s way of making sure that those functions can operate orderly and safely. Also I love the modular setup your team has taken to pipeline building!
👍 2
c

Chris White

08/05/2019, 9:23 PM
@Marvin archive “How to organize Flows?”
5 Views