Maybe this is a dask questions. I'll admit I have ...
# prefect-community
r
Maybe this is a dask questions. I'll admit I have not properly investigated this much (today's job) but just looking for ideas
i
@Rob Fowler Not sure if this is what you are after but prefect allows you to set concurrency limits
j
@Rob Fowler indeed, Prefect Cloud’s enterprise tiers include flexible task concurrency limits that can be enforced globally (across all clusters / flows) and are based on task
tags
.
We will be bringing those to lower tiers of Cloud in the near future
r
this is for a POC, we don't have prefect in production yet. I am sure if it works out (we have other use cases) I'll be looking at some license, just not cloud as we are a cloud provider ourselves.
just back from dropping the kids off. So I see I can tag the task then set a task limit with
Copy code
update_task_tag_limit
awesome. Out of the box functionality.
j
Yes, as a matter of fact I believe you’ll want to call
update_task_concurrency_limit
- the other one will work but is a deprecated route.
r
in the docs, the tag one is described, no notes on it being depreciated yet
finally got back to this, seems it's cloud only, back to the dask idea. This won't work in cloud as it needs to connect to winrm hosts
Copy code
executor = LocalDaskExecutor(pool=ThreadPool(), num_workers=N)
Not thanking you for saying in the doco that kwargs are Dask config arguments when actually you also mess with num_workers independently. So:
Copy code
executor = LocalDaskExecutor(pool=ThreadPool(N)
as expected does not work.