@Marvin Since a few days, in prefect cloud, our scheduled flows remain stuck in the “Scheduled” state even after the execution time has passed. Previously, these flows correctly transitioned to the “Late” state, triggering our VM startups. I already tried: • Rescheduled the flows. • Deactivated and reactivated schedules. Anything else we can do?

It seems you're experiencing an issue where scheduled flows in Prefect Cloud remain stuck in the "Scheduled" state instead of transitioning to "Late". This can be quite frustrating, I imagine. Based on the recent findings, here are some steps you can take to troubleshoot this issue: 1. Check Worker Configuration: Ensure that your workers are running and properly configured to pick up scheduled flows. Misconfigurations or network issues can prevent workers from retrieving scheduled runs. 2. Review Scheduler Settings: The Prefect Scheduler only creates new flow runs and places them in the Scheduled state. It doesn't handle execution. Make sure that your scheduling settings are correct and that the scheduler is functioning as expected. You can adjust scheduler behaviors through the Prefect config settings. Learn more about scheduler settings. 3. Inspect Logs: Look at the logs for any errors or warnings that might indicate why the flows are not transitioning. This might provide clues about what's going wrong. 4. Update Prefect: If you haven't already, consider updating to the latest version of Prefect. Sometimes, bugs are fixed in newer releases. 5. Investigate Known Issues: There are known issues where flows can get stuck in various states due to configuration or deployment issues. You might want to explore some of these reported issues on GitHub to see if any align with your problem: - Flow runs stuck in AwaitingRetry after deployment deletion - Runs in AwaitingConcurrencySlot not picked up If these steps don't resolve your issue, you might consider reaching out to Prefect support for more targeted assistance. They may have additional insights or solutions tailored to your specific setup.

@Marvin This is in prefect cloud, we cannot update it. And it is not related to our local code, as they scheduling happens on prefect cloud and the states are managed their independently of our local code (a flow should transition into late when the execution time passes).

Given that you're using Prefect Cloud and the issue is related to the scheduling and state management handled by the cloud service, here are a few additional suggestions: 1. Check Prefect Cloud Status: Verify if there are any known issues or maintenance activities occurring on Prefect Cloud that might affect scheduling. Prefect's status page or support channels might have relevant information. 2. Contact Prefect Support: Since this involves Prefect Cloud's internal scheduling, reaching out to Prefect support might be your best bet. They can provide insights specific to your account and diagnose backend issues. 3. Verify Deployment and Agent Setup: Ensure your deployments and agents in Prefect Cloud are correctly configured. Sometimes, issues with agent connectivity or deployment settings can affect flow execution. 4. Review Concurrency and Limits: Check if there are any concurrency limits or other configurations that might be preventing the flows from transitioning as expected. These steps should help you identify or resolve the issue with Prefect Cloud's scheduling. If the problem persists, Prefect support will likely have more direct access to diagnose and address the issue.

@Marvin Everything worked fine for months, and it suddenly stopped working, so I suspect a bug in their cloud. I will contact support!

That sounds like a reasonable course of action. Given the sudden change in behavior, contacting Prefect support will likely provide the quickest resolution, especially if it's related to a bug or a change in their cloud infrastructure. Best of luck, and I hope they can resolve the issue swiftly for you. If you have any more questions or need further assistance, feel free to reach out.

@Marvin Here can you see it:

It seems like you were about to share something, but the message got cut off. If you have additional details, links, or screenshots to share, please feel free to provide them. I'll do my best to assist you with the information you provide.

@Marvin there is a screenshot attached to the previous message.

Unfortunately, I can't view images or screenshots. However, if you can describe the content of the screenshot or any error messages, logs, or details it contains, I'd be happy to help interpret or provide guidance based on that information.