Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
g
gperrone
05/03/2023, 5:52 PMIs there a limit to how many tasks I can create in Prefect? I am trying to submit a total of~4k tasks and it gets to about 3.3K of them before getting a
Crash detected! Execution was cancelled by the runtime environmentMaybe I am missing something but is there a way to have Prefect only create a certain amount of tasks at a time before creating the rest?
I was able to surface a somwhat more helpful eerror log:
Traceback (most recent call last):
File "/Users/gperrone/.pyenv/versions/3.8.12/lib/python3.8/contextlib.py", line 189, in __aexit__
await self.gen.athrow(typ, value, traceback)
File "/Users/gperrone/.pyenv/versions/dasa-etl/lib/python3.8/site-packages/prefect/task_runners.py", line 165, in start
yield self
File "/Users/gperrone/.pyenv/versions/dasa-etl/lib/python3.8/site-packages/prefect/engine.py", line 388, in begin_flow_run
terminal_or_paused_state = await orchestrate_flow_run(
File "/Users/gperrone/.pyenv/versions/dasa-etl/lib/python3.8/site-packages/prefect/engine.py", line 669, in orchestrate_flow_run
result = await flow_call.aresult()
File "/Users/gperrone/.pyenv/versions/dasa-etl/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 181, in aresult
return await asyncio.wrap_future(self.future)
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
KeyError: 13033b
Bianca Hoch
05/03/2023, 8:25 PMHey Giancarlo, thanks for raising this. Are you self-hosting or using Prefect cloud? If you are using Prefect Cloud, there is a flow and task creation limit of 400 runs/minute which is applied to personal accounts. You should be receiving a
429g
gperrone
05/03/2023, 8:52 PMYeah I am currently using Prefect cloud. I don’t see any 429 response codes in the logs however, and the other flow that I have runs 1.2k tasks concurrently with no 429 error either
b
Bianca Hoch
05/04/2023, 5:21 PMWhat infrastructure are you using to run your tasks? Could be that some form of resource contention is happening there