Hi all ! We're currently fighting with mapping features, which overflow the backend when mapping over many inputs (which is a requirement for our flows - typically over 50) and cause the flow to crash. When running locally, the flow crashes with a message that the database is locked (on the example below) - when running on the server, we do get an error 500 back after many creations in a row (

Crash detected! Execution was cancelled by the runtime environment.

and then

Crash detected! Execution was interrupted by an unexpected exception: httpx.HTTPStatusError: Server error '500 Internal Server Error' for url 'http://[our instance url]/api/task_runs/'

For more information check: <https://httpstatuses.com/500>

) A minimum running example is (in Python 3.10 with prefect 2.7.9) :

import time
from prefect import unmapped, flow, task

@task(name="Sleeper {id}")
def sleeper(id, seconds_to_wait: int):
    print(f"Start Sleeper {id}")
    time.sleep(seconds_to_wait)
    print(f"End Sleeper {id}")

@flow(
    name="Prefect 2 concurrency checker",
)
def flow(n_runs: int = 128, seconds_to_wait: int = 2):
    res = sleeper.map(list(range(n_runs)), seconds_to_wait=unmapped(seconds_to_wait))
    return res


if __name__ == "__main__":
    res = flow()
    print("done")

In Prefect 1, we had a way to limit the number of task run creations, but that's apparently not available in Prefect 2 AFAIK (note: we've tried tag-limiting, but that's not helping as it's the creation of the many tasks that kills it, not their actual run ; we've also tried an async approach, which fails in the same way). My questions to you: 1- Have you ever experienced this too? 2- Did you find a walk-around? 3- If not, do you have an idea of how to manage this?

Hey, we experienced the same behavior when deploying on a server. The flow runs well with a mapping over a small number of tasks (e.g. 200) but if the number scales up it crashes randomly with the 500. Could not figure out how to solve this.

Best solutions we found so far: • Disable JIT on your database to increase performance - been fixed in a recent Prefect update, so you might not have to anymore. • Increase DB timeouts in the Prefect configuration. • Give the Prefect DB more resources (CPU, mem, etc.), we run it as a virtual server. • Btw. use some monitoring tool that can tell you the load on the DB, we use netdata. Can help determine if it can keep up. • Be aware if you're doing a lot of logging, it can also increase load on the database. Enabling the pg_stat_statements table can be useful in determining this.

We still have this issue with 2.8.7. It crashes randomly when submitting a large amount of tasks (crash due the creation).