Thread
#prefect-community
    nick vazquez

    nick vazquez

    1 year ago
    I was curious what was the limit for the number of mapped tasks? (not sure if I have the terms correct) I want to run some adhoc flows that potentially kick off anywhere between 100-100,000 tasks (the same task but parameterized)
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @nick vazquez, I think the answer for this is the limits of your hardware. We have seen workloads on the order of 10,000 task runs.
    nick vazquez

    nick vazquez

    1 year ago
    (Nice talk at the dask summit btw ๐Ÿ™‚) Thanks for the quick response! -- when you say
    limits of hardware
    , you are referring to amount of memory to hold the intermediate values between dask tasks?
    Kevin Kho

    Kevin Kho

    1 year ago
    Thank you for watching! Yes and also if your cluster has enough workers to run the tasks.
    nick vazquez

    nick vazquez

    1 year ago
    Does it need to be able to run all of the mapped tasks, or will it just queue them?
    Kevin Kho

    Kevin Kho

    1 year ago
    It will queue them in a depth first execution manner (although sometimes do it in breadth first execution, itโ€™s hard to control). If you have 1000 tasks over 10 workers, it will run 10 at a time.
    nick vazquez

    nick vazquez

    1 year ago
    perfect! Thank you!! I also had some other questions (I hope they're quick) that I put in another channel https://prefect-community.slack.com/archives/C014Z8DPDSR/p1622077683237700
    Kevin Kho

    Kevin Kho

    1 year ago
    Yeah I was looking at this. Will respond there.