Lior Barak
07/15/2023, 3:19 PMEmil Christensen
07/17/2023, 4:21 PMDaskTaskRunner
constructor (docs):
# Use 4 worker processes, each with 2 threads
DaskTaskRunner(
cluster_kwargs={"n_workers": 4, "threads_per_worker": 2}
)
Also, if you’re just making http calls, you don’t have to use Dask, you could use the default ConcurrentTaskRunner which executes submitted tasks using Python’s native concurrency. Should be lighter weight if you are only IO bound.Lior Barak
07/18/2023, 7:44 AMEmil Christensen
07/18/2023, 2:52 PM.submit
?Lior Barak
07/27/2023, 9:57 AMsubmit
properly
is there an upper limit for ConcurrentTaskRunner
?
lets say I want to run 1000 flows with 100 tasks each in parallel (on Server not cloud)
can I just make Prefect run on a strong machine?