Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Maybe this is a dask questions. I'll admit I have not properly investigated this much (today's job) but just looking for ideas

<@U01A7FF0NL8> Not sure if this is what you are after but prefect allows you to <https://docs.prefect.io/orchestration/concepts/task-concurrency-limiting.html#setting-concurrency-limits|set concurrency >limits

<@U01A7FF0NL8> indeed, Prefect Cloud’s enterprise tiers include flexible task concurrency limits that can be enforced globally (across all clusters / flows) and are based on task `tags`.

We will be bringing those to lower tiers of Cloud in the near future

this is for a POC, we don't have prefect in production yet. I am sure if it works out (we have other use cases) I'll be looking at some license, just not cloud as we are a cloud provider ourselves.

just back from dropping the kids off. So I see I can tag the task then set a task limit with
```update_task_tag_limit```

Yes, as a matter of fact I believe you’ll want to call `update_task_concurrency_limit` - the other one will work but is a deprecated route.

in the docs, the tag one is described, no notes on it being depreciated yet

finally got back to this, seems it's cloud only, back to the dask idea. This won't work in cloud as it needs to connect to winrm hosts

```executor = LocalDaskExecutor(pool=ThreadPool(), num_workers=N)```
Not thanking you for saying in the doco that kwargs are Dask config arguments when actually you also mess with num_workers independently.
So:
```executor = LocalDaskExecutor(pool=ThreadPool(N)```
as expected does not work.