hi! i am self hosting prefect (orion as well as the flows) in kubernetes cluster . i used https://github.com/PrefectHQ/prefect-helm to setup my orion server and https://github.com/anna-geller/dataflow-ops-aws-eks/blob/main/.github/workflows/eks_prefect_agent.yml for deploying agents. im using kubernetes job block as the infrastructure block and s3 block for the storage. My flows have high memory usage compared to cpu usage ( tending to 1-2 cores and 12gb memory) im running 5 flows at a time with 2000-3000 batches of 50 tasks and running 1000s of flows throughout the day . i have implemented task concurrency limits and flow concurrency limits on every work queue. Im consistently experiencing flow run crashes . Besides concurrency limits , what options do i have to control the load and/or are there any methods to control the resource usage of the jobs in my cluster?

There are work queue concurrency limits for flow runs and task concurrency limits for tasks You can try Karpenter to help scale EKS cluster more easily

ive implemented both of the limits.

What kind of crashes are you seeing?

Flow run infrastructure exited with non-zero status code -1.

generates no other logs in agent or the job pod PS: it also leaves the tasks running thus filling up the concurrency limit with orphan tasks

Hm. Do you have DEBUG level logs on?

PREFECT_LOGGING_LEVEL: INFO
PREFECT_LOGGING_SERVER_LEVEL: WARNING
PREFECT_LOGGING_SETTINGS_PATH: ${PREFECT_HOME}/logging.yml
PREFECT_LOGGING_EXTRA_LOGGERS: 
PREFECT_LOGGING_LOG_PRINTS: false

this is the current config. i can try changing it

I think -1 usually means that we could not monitor the job

i havent configured this myself. it must be the default value

The default value changed to fix this issue; what version are you on?

im using 2.7.1

Great! Let us know if you need anything else

we have same issue, prefect is 2.7.1, having random crashes with

Flow run infrastructure exited with non-zero status code -1

.

actually i have noticed this error in 2 cases. 1. when we were not scheduling jobs to go to separate nodes and they all got scheduled on the same node, exhausting the resources for it and thus crashing.Adding podaffinity and antiaffinity helped with this 2. when we run around 6000-7000 tasks in batches of 50 the ui starts to lag , the mini representation of radar also lags and shows crashed (in this case the job somehow keeps running in the background using all resources accordingly). i need some assistance with the second case because this is unexpected behaviour. if it crashes it should stop running in the background

in our case was non of this, job_ and pod_ timeouts ware the issue. We set to very long number and the issue was gone. Like Michael have told above, confusion is 2.7.1 fixes the issue but it does not. You have to manually set values otherwise after 60 seconds randomly majority of scheduled jobs would be marked as crashed.