hey peeps, any preferences for organizing projects/namespaces based on prefect flows?

What exactly do you mean by projects/namespaces?

oh just like a generic way of organizing files

Oh yeah great question! Locally flows are simply importable objects so you can really organize them any way you would like. Once flows are serialized and deployed to cloud you are able to organize them into

projects

which can contain as many flows (and as many versions of those flows) as you would like. Projects in this sense act as a folder structure.

ok cool thx!

Here is how we are organizing ourselves: We defined three submodules as follows:

xxx
    functions
    tasks
    flows

The

xxx

module is our “data pipeline package”. The

xxx.functions

module has the low-level functions or scripts that do most of the important work. These are written without any prefect knowledge. An example function: bandpass filter a file with some signals. The

xxx.tasks

has objects that derive from

prefect.Task

and manage the inputs, outputs, parameters of one or many functions in the

xxx.functions

module. The usage of these functions gives some semantic meaning to the task. So far for us, the line between task and function is still blurry, but sometimes it makes sense to group several functions together to avoid sending a lot of data between workers. The

xxx.flows

then orchestrates several tasks defined in

xxx.tasks

. It connects them and manages how input/outputs are connected, how failures are managed, logging, etc.

@David Ojeda Yeah in a way tasks and functions can be quite similar. The best way to think about it is functions do something while tasks are Prefect’s way of making sure that those functions can operate orderly and safely. Also I love the modular setup your team has taken to pipeline building!

@Marvin archive “How to organize Flows?”