I'm running into a case where the prefect cloud UI shows jobs as submitted. I have queried the graphql api by the flow id to confirm it is submitted (i presume that's where the ui gets it's info from anyway). The agent however can finds no ready flows. I am running with debug mode to see that it's polling and never finding anything:

[2022-07-14 17:49:44,859] DEBUG - agent | Querying for ready flow runs...
DEBUG:agent:Querying for ready flow runs...
[2022-07-14 17:49:44,959] DEBUG - agent | No ready flow runs found.
DEBUG:agent:No ready flow runs found.
DEBUG:agent:Sleeping flow run poller for 10.0 seconds...
[2022-07-14 17:49:44,960] DEBUG - agent | Sleeping flow run poller for 10.0 seconds...
[2022-07-14 17:49:54,960] DEBUG - agent | Querying for ready flow runs...
DEBUG:agent:Querying for ready flow runs...
[2022-07-14 17:49:55,046] DEBUG - agent | No ready flow runs found.
DEBUG:agent:No ready flow runs found.
[2022-07-14 17:49:55,046] DEBUG - agent | Sleeping flow run poller for 10.0 seconds...
DEBUG:agent:Sleeping flow run poller for 10.0 seconds...

Anyone have an suggestions on how to continue to debug?

check suggestions in this thread on Discourse https://discourse.prefect.io/t/why-is-my-flow-stuck-in-a-submitted-state/201

Thank you! I did see that kind of documentation, but it was confusing because I wasn't seeing any action in my new kubernetes cluster. My issue was that we have just recently deployed a new kubernetes cluster and we left the prefect agent running in the old cluster. As a result some jobs would get picked up by the polling by the agent in the old cluster, and ultimately fail. Turning off the agent in the old cluster has fixed the issue.

nice work figuring out the old cluster was the issue 🙌