https://prefect.io logo
Title
g

gperrone

05/03/2023, 5:52 PM
Is there a limit to how many tasks I can create in Prefect? I am trying to submit a total of~4k tasks and it gets to about 3.3K of them before getting a
Crash detected! Execution was cancelled by the runtime environment
error log with no other context. What is everyone’s experience with submitting a large amount of tasks? I have a different flow that submits about 1000 concurrent tasks and I have not had any problems with that one
Maybe I am missing something but is there a way to have Prefect only create a certain amount of tasks at a time before creating the rest?
I was able to surface a somwhat more helpful eerror log:
Traceback (most recent call last):
  File "/Users/gperrone/.pyenv/versions/3.8.12/lib/python3.8/contextlib.py", line 189, in __aexit__
    await self.gen.athrow(typ, value, traceback)
  File "/Users/gperrone/.pyenv/versions/dasa-etl/lib/python3.8/site-packages/prefect/task_runners.py", line 165, in start
    yield self
  File "/Users/gperrone/.pyenv/versions/dasa-etl/lib/python3.8/site-packages/prefect/engine.py", line 388, in begin_flow_run
    terminal_or_paused_state = await orchestrate_flow_run(
  File "/Users/gperrone/.pyenv/versions/dasa-etl/lib/python3.8/site-packages/prefect/engine.py", line 669, in orchestrate_flow_run
    result = await flow_call.aresult()
  File "/Users/gperrone/.pyenv/versions/dasa-etl/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 181, in aresult
    return await asyncio.wrap_future(self.future)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

KeyError: 13033
b

Bianca Hoch

05/03/2023, 8:25 PM
Hey Giancarlo, thanks for raising this. Are you self-hosting or using Prefect cloud? If you are using Prefect Cloud, there is a flow and task creation limit of 400 runs/minute which is applied to personal accounts. You should be receiving a
429
response in the logs if that is the case.
g

gperrone

05/03/2023, 8:52 PM
Yeah I am currently using Prefect cloud. I don’t see any 429 response codes in the logs however, and the other flow that I have runs 1.2k tasks concurrently with no 429 error either
b

Bianca Hoch

05/04/2023, 5:21 PM
What infrastructure are you using to run your tasks? Could be that some form of resource contention is happening there