I was curious what was the limit for the number of mapped tasks? (not sure if I have the terms correct) I want to run some adhoc flows that potentially kick off anywhere between 100-100,000 tasks (the same task but parameterized)

Hey @nick vazquez, I think the answer for this is the limits of your hardware. We have seen workloads on the order of 10,000 task runs.

(Nice talk at the dask summit btw 🙂) Thanks for the quick response! -- when you say

limits of hardware

, you are referring to amount of memory to hold the intermediate values between dask tasks?

Thank you for watching! Yes and also if your cluster has enough workers to run the tasks.

Does it need to be able to run all of the mapped tasks, or will it just queue them?

It will queue them in a depth first execution manner (although sometimes do it in breadth first execution, it’s hard to control). If you have 1000 tasks over 10 workers, it will run 10 at a time.

perfect! Thank you!! I also had some other questions (I hope they're quick) that I put in another channel https://prefect-community.slack.com/archives/C014Z8DPDSR/p1622077683237700

Yeah I was looking at this. Will respond there.