Hi all ! We're currently fighting with mapping fe...
# prefect-community
Hi all ! We're currently fighting with mapping features, which overflow the backend when mapping over many inputs (which is a requirement for our flows - typically over 50) and cause the flow to crash. When running locally, the flow crashes with a message that the database is locked (on the example below) - when running on the server, we do get an error 500 back after many creations in a row (
Crash detected! Execution was cancelled by the runtime environment.
and then
Crash detected! Execution was interrupted by an unexpected exception: httpx.HTTPStatusError: Server error '500 Internal Server Error' for url 'http://[our instance url]/api/task_runs/'
For more information check: <https://httpstatuses.com/500>
) A minimum running example is (in Python 3.10 with prefect 2.7.9) :
Copy code
import time
from prefect import unmapped, flow, task

@task(name="Sleeper {id}")
def sleeper(id, seconds_to_wait: int):
    print(f"Start Sleeper {id}")
    print(f"End Sleeper {id}")

    name="Prefect 2 concurrency checker",
def flow(n_runs: int = 128, seconds_to_wait: int = 2):
    res = sleeper.map(list(range(n_runs)), seconds_to_wait=unmapped(seconds_to_wait))
    return res

if __name__ == "__main__":
    res = flow()
In Prefect 1, we had a way to limit the number of task run creations, but that's apparently not available in Prefect 2 AFAIK (note: we've tried tag-limiting, but that's not helping as it's the creation of the many tasks that kills it, not their actual run ; we've also tried an async approach, which fails in the same way). My questions to you: 1- Have you ever experienced this too? 2- Did you find a walk-around? 3- If not, do you have an idea of how to manage this?