https://prefect.io logo
Title
r

Rob Fowler

09/09/2020, 9:47 PM
Maybe this is a dask questions. I'll admit I have not properly investigated this much (today's job) but just looking for ideas
i

itay livni

09/09/2020, 9:52 PM
@Rob Fowler Not sure if this is what you are after but prefect allows you to set concurrency limits
j

Jeremiah

09/09/2020, 10:27 PM
@Rob Fowler indeed, Prefect Cloud’s enterprise tiers include flexible task concurrency limits that can be enforced globally (across all clusters / flows) and are based on task
tags
.
We will be bringing those to lower tiers of Cloud in the near future
r

Rob Fowler

09/09/2020, 11:03 PM
this is for a POC, we don't have prefect in production yet. I am sure if it works out (we have other use cases) I'll be looking at some license, just not cloud as we are a cloud provider ourselves.
just back from dropping the kids off. So I see I can tag the task then set a task limit with
update_task_tag_limit
awesome. Out of the box functionality.
j

Jeremiah

09/09/2020, 11:13 PM
Yes, as a matter of fact I believe you’ll want to call
update_task_concurrency_limit
- the other one will work but is a deprecated route.
r

Rob Fowler

09/09/2020, 11:17 PM
in the docs, the tag one is described, no notes on it being depreciated yet
finally got back to this, seems it's cloud only, back to the dask idea. This won't work in cloud as it needs to connect to winrm hosts
executor = LocalDaskExecutor(pool=ThreadPool(), num_workers=N)
Not thanking you for saying in the doco that kwargs are Dask config arguments when actually you also mess with num_workers independently. So:
executor = LocalDaskExecutor(pool=ThreadPool(N)
as expected does not work.