Thread
#prefect-server
    Sylvain Hazard

    Sylvain Hazard

    11 months ago
    Hello there ! I'd like to know about any limitation as to how large a mapped task can be. This does not have to be a fixed limit, an order of magnitude would do. Given an input list of size, let's say 1 million, would Prefect Server be able to map a task correctly ? I am running said flow on a LocalDaskExecutor. I believe it is relevant due to it going for a depth first execution. Thanks a bunch !
    Anna Geller

    Anna Geller

    11 months ago
    Hi @Sylvain Hazard, Iโ€™d say it depends to some extent on your database and infrastructure setup (e.g. DaskExecutor on Dask distributed can handle way more mapped tasks than a non-distributed LocalDaskExecutor). I will ask the team if they can tell you something more concrete, and get back to you. But you would probably get the most realistic number if you would run a test yourself on your DEV infrastructure.
    Sylvain Hazard

    Sylvain Hazard

    11 months ago
    Seems like the answer I was looking for : it entirely depends on the execution cluster and there is no limit induced by the prefect core API, is that right ? Given unlimited scaling capabilities on a Dask cluster, I should be able to run an infinitely large map ?
    Anna Geller

    Anna Geller

    11 months ago
    correct, that what I would expect. However, Prefect needs to track the state of all those mapped tasks, which means it needs to register those child tasks in the database. So the database itself could be more of a limit than Dask. I will ask the team because Iโ€™m giving you a vague answer.
    Sylvain Hazard

    Sylvain Hazard

    11 months ago
    The database is limiting only in a storage capacity sense as well as registering time though, right ? I'd expect to only be in trouble if registering a number of tasks that would fill my storage.
    Anna Geller

    Anna Geller

    11 months ago
    correct, afaik there is some queueing mechanism to batch-register child tasks, but to be honest I donโ€™t know (yet) how it works exactly
    Sylvain Hazard

    Sylvain Hazard

    11 months ago
    Don't worry, my question was quite vague too ๐Ÿ˜… It was more of a question to quench my curiosity as well as vaguely remembering a mention to this in the DFE article on Medium.
    Anna Geller

    Anna Geller

    11 months ago
    Your question was well-justified. Distributed computing is hard ๐Ÿ™‚
    Sylvain Hazard

    Sylvain Hazard

    11 months ago
    Definitely is, that's why I am starting easy with a LocalDaskExecutor on a single node ๐Ÿ˜„
    Kevin Kho

    Kevin Kho

    11 months ago
    Yeah this is mainly about the limits of your hardware and what the mapped task is doing. This was not on server, but I was testing mapping a list of 400,000 items on Prefect Cloud on a Local Agent on my laptop. The task barely just returned
    x + 1
    and it worked. From a registering standpoint, I think you need batching at 10000 tasks (but mapped tasks count as 1 for registration). The runtime will generate multiple tasks though and yes this can fill your storage. I think the thing to note though is that a lot of users unnecessarily do this. If you have a very high mapped task count, just make sure that you do need observability and retries for each of those individual elements. Otherwise, you might be able to do the operation with a DataFrame.