@Marvin How does Concurrency limits handle scheduled jobs? I have no flows running with a work pool concurrency limit of 3. I launch 2 jobs and both get stuck in a

Late

state. I remove the concurrency limit and suddenly those jobs kick off

It seems that your issue with jobs getting stuck in a "Late" state under a concurrency limit might be related to how concurrency slots are managed and refreshed. Here are some insights: 1. Concurrency Limits and Slot Decay: When you set a concurrency limit, each slot becomes unavailable once occupied and only becomes available again over time based on the slot decay rate. If the decay rate is slow, it could cause delays in starting new jobs if slots are not refreshing quickly enough. You can configure the

slot_decay_per_second

parameter to adjust how quickly slots refresh. Learn more about concurrency limits and slot decay here. 2. Scheduler Behavior: The Prefect Scheduler is responsible for creating new flow runs and placing them in a "Scheduled" state. It does not handle the execution of these runs. If the concurrency limit is too restrictive, it might prevent scheduled jobs from transitioning to an active running state. Read more about how the scheduler works here. 3. Late State: The "Late" state is a specific state indicating that a flow run was scheduled to start but did not begin execution on time. This can occur if there are insufficient resources or slots available due to concurrency limits. 4. Potential Issues: There might be underlying issues reported in the Prefect GitHub repository related to concurrency limits and scheduling. However, no specific issue was found directly related to your scenario. To resolve the issue, consider adjusting the slot decay rate or reviewing your concurrency configuration to ensure it aligns with your execution needs. If the problem persists, it might be worth exploring any open issues on the Prefect GitHub repository or reaching out to the Prefect community for further assistance.

@Marvin do Scheduled jobs count against the concurrency limit?

Scheduled jobs do not count against the concurrency limit. Concurrency limits are applied when tasks or flows are actively running, not when they are merely scheduled. The Prefect Scheduler creates flow runs and places them in a "Scheduled" state, but these do not occupy concurrency slots until they transition to an active running state. For more details, you can explore the Prefect documentation on concurrency limits.