Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

When an deployment gets kicked off to a pool with two agents, both agents should be able to do the deployment whenever there are concurrent tasks in the deploy right? For some reason I have a job with 100 concurrent tasks and 10 agents, but only 1 agent is doing the work right now

mm that's now how agents work. a flow is ultimately just a python script, and tasks aren't distributed across multiple runtimes or hardware unless you're doing so intentionally with something like an external dask cluster

if your deployment has the process infrastructure type, only one agent is going to do the work, and it's going to pick up that flow run and execute the entire thing

Thansk <@U05TQMN8RUP>! This applies to agents and workers right? In order to have distributed compute, I have to break up the tasks into deployments and then run it that way?

OR maybe, the better question is,  what's the set up like with External dask clusters? Im still trying to get used to Prefect, sorry for the multiple questions