jpuris
09/20/2022, 8:26 AM10
but are plenty, when it is left to Unlimited?
When there are no visible scheduled jobs, my agent polling that queue would not retrieve any jobs either 😕 As soon as I change it to Unlimited
, it comes to life and starts running all the late jobs like nothing had happened.
I’m using Prefect Cloud v2Mason Menges
09/20/2022, 4:32 PMjpuris
09/20/2022, 4:40 PMprefect deployment build …
2. Some edits to the yaml e.g. schedule changed to interval
3. prefect deployment apply …
All was well until the next day (I had shut down the agent for the night) and hence quite many schedules were late.
4. Started up agent and shut it down soon after (it started to replay all of late schedules)
5. Changed the flow concurrency in the queue (In UI) to 10
6. Started up agent and it run for a while after which I noticed no new logs rolling in
7. Queue on Prefect UI is empty.. this is where I started to look into the concurrency and discovered that toggling it off (set to Unlimited) brought back the expected functionality.Mason Menges
09/20/2022, 4:46 PMjpuris
09/21/2022, 6:43 AMUnlimited
“Flow Run Concurrency (Optional)” setting to 10 and 20 and the problem is there as described earlier, but if I change it to 30, I can see 10 schedules with setting of 40, I see 20 schedules.
I suppose you’d still require steps to reproduce this sort of situation?Jenny
09/21/2022, 6:31 PMjpuris
09/21/2022, 7:26 PMJenny
09/21/2022, 7:37 PMjpuris
09/21/2022, 7:39 PMRio McMahon
09/27/2022, 3:36 PMJenny
09/27/2022, 3:53 PMRio McMahon
09/27/2022, 3:55 PMScheduled
to Late
. They are running on a 1 minute cadence. The agent polling for the queue never picks them up (confirming by looking at agent logs). Flows just continually pile up as Late
without any ever being run.Jenny
09/27/2022, 3:58 PMRio McMahon
09/27/2022, 4:03 PMJenny
09/27/2022, 4:04 PM