Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

*Multiple Agents for Disparate Projects?*

Given several disparate projects, each with its own dependencies (installed packages and custom python modules), does it make sense to run:

A. One agent pre project

or

B. One agent to service all the projects?

Hi <@U01FXDRMC4E> - either model is fine but I'd recommend the latter where your agents serve more general service patterns (resources, environments etc), assuming your flows have disparate requirements, rather than having a single agent.

Do you mean "if the resources and environments are similar, use a single agent"?

Well it depends, you might separate your agents by environment - for example dev and prod - where the agent running in prod is on a large machine with lots of resources and uptime checks etc and the dev one less so - so you could then label flows (or specific runs of a flow!) based on where you want something to run

It boils down to: there's not really a correct answer, it's what makes the most sense for your infrastructure and the types of requirements your flows have

I'd say to start with a single agent and scale out as needed