https://prefect.io logo
Title
а

Андрій Демиденко

04/06/2022, 1:18 PM
Hi everybody, It seems like our team is going to have tons of flows. Is there any ideas how to create a hierarchy of the flows or to arrange them somehow? Thank you.
:discourse: 1
a

Anna Geller

04/06/2022, 1:29 PM
Hard to tell without knowing your use case. It depends a lot on how you would like to structure your flows. E.g. if you would build it for a data warehousing project, you could structure your flows according to your data warehouse schemas:
dwh_project
|-- business_vault
|-- data_mart
|-- data_services
|-- monitoring
|-- operational
|-- raw_vault
`-- stage
    |-- crm
    |-- erp
    |-- flat_file
    |-- google_analytics
    |-- price_api
    |-- shipment_api
    |-- shop_system
    `-- wms
I used a similar structure as the above in my last DWH project. But if you need a repo structure for multiple teams, you could build a structure around your teams:
teams_flows
|-- analytics_engineering
|-- automations
|-- data_engineering
|   |-- data_ingestion_flows
|   `-- data_replication_flows
|-- data_science
|   |-- batch_prediction_flows
|   |-- preprocessing_flows
|   `-- training_flows
`-- platform
It's super hard to give any recommendations without knowing your use case
and you could also opt for a single repo per project - e.g. having all ML training flows within their own repository and have a structure based on their business logic
а

Андрій Демиденко

04/06/2022, 1:32 PM
I'm looking for a tool how to separate some flows into kind of a group inside a project(it seems like we are going to have 1 repo but still tons of flows)
a

Anna Geller

04/06/2022, 1:34 PM
can you explain your use case? there is no tool that will build a structure for your repo; it's an important decision you need to make based on your use case and the needs of your organization, not based on some tool
а

Андрій Демиденко

04/06/2022, 1:38 PM
For example we have a project "working_with_client" but we would also want to have some folder "client_name" containing a group of flows related to the specific client, which we can see in the UI and if it's possible to give some partial permissions to this (but it's optional)
a

Anna Geller

04/06/2022, 1:40 PM
I see, so it's not about a structure of a repo but rather a way to provide permissions based on a project, correct? are you asking for Prefect 2.0 or 1.0?
а

Андрій Демиденко

04/06/2022, 1:41 PM
1.0
a

Anna Geller

04/06/2022, 1:42 PM
can you answer all my questions rather than only the last one? 🙂
а

Андрій Демиденко

04/06/2022, 1:43 PM
Yes the first is correct (I didn't notices the question 🙂 )
a

Anna Geller

04/06/2022, 1:43 PM
thanks a lot! are you on Prefect Cloud or on Prefect Server?
а

Андрій Демиденко

04/06/2022, 1:43 PM
Cloud
a

Anna Geller

04/06/2022, 1:48 PM
There's a way of providing permissions based on a project, but I believe this is only for Enterprise plans. Afaik such granular permissions are not available on a self-serve Cloud tier. The only restriction you could impose on a self-serve plan is providing Read-only access - perhaps that's enough for your use case? Otherwise, you can send an email to sales@prefect.io to get more detailed information