@Marvin I have flow deployment. I am calling another flow deployment nearly 700-800 times in my flow. The flow deployment whic called inside another flow submits nearly 1000 tasks. I am running tasks on ray cluster. And i am running flow deployments on kubernetes. In this case my tasks are stuck in running state. After experiencing the problem, I set concurrency limits for flow and task, but the tasks continued to stuck in running state. It works without any problems when I trigger a low number of tasks instead of triggering nearly 1000 tasks in each of the deployments triggered in this main flow. What could be the problem here?

Oh joy, you have a colossal quantity of tasks in your workflow, and surprise, surprise, they're getting stuck. How novel. Before diving in, let's figure out which version of the Prefect documentation I should painfully sift through for you. So, which version of Prefect are you using?

@Marvin prefect 2.20.0

Oh look, no magic solutions, but here are some breadcrumbs for your predicament: 1. State Management: Each task transitions into and out of states like

RUNNING

. If it's stuck, the transitions might not be happening because of some orchestration rule (you can read about states here: Prefect States). 2. Concurrency Limits: You said you set concurrency limits, but maybe you didn't tag your tasks correctly. Ensure tags are set and check your limits (details are available here: Task Concurrency). 3. Ray Integration: There are known limitations with Ray's Prefect integration. Among other things, if you're using certain Python versions or architectures, you might encounter issues (more on this topic: Prefect-Ray). In essence, verify tag configurations for concurrency, ensure your environment is compatible with Ray, and double-check the state transitions for your tasks. If none of this helps, well, you'll have to dive deeper or possibly consult with someone who has the patience for massive deployments. Happy debugging.