Hey there! One of the work pools of Prefect Cloud stopped sending the flows yesterday for unknown re...

Dmitrii Egunov

04/06/2024, 4:59 PM

Hey there! One of the work pools of Prefect Cloud stopped sending the flows yesterday for unknown reason. The logs are just polling:

Copy code

06 April 2024 at 18:34 (UTC+2:00)	16:34:49.307 | DEBUG | prefect.agent - Checking for scheduled flow runs...	prefect
06 April 2024 at 18:34 (UTC+2:00)	16:34:44.917 | DEBUG | prefect.agent - Checking for cancelled flow runs...	prefect
06 April 2024 at 18:34 (UTC+2:00)	16:34:33.683 | DEBUG | prefect.agent - Checking for scheduled flow runs...	prefect
06 April 2024 at 18:34 (UTC+2:00)	16:34:17.222 | DEBUG | prefect.agent - Checking for scheduled flow runs...	prefect
06 April 2024 at 18:34 (UTC+2:00)	16:34:06.982 | DEBUG | prefect.agent - Checking for cancelled flow runs...	prefect

The status is correctly green - I’ve tried to stop the ECS service that runs the agent and it went red. I’ve also tried to start the work pool locally and patch the Cloud querying with additional debug logs - it is indeed returns empty responses. There were no deployments or changes on our side and it’s quite bizarre. Agent version is v2.14.3. What could be the problem? cc @Prefect

✅ 1

Nate

04/06/2024, 10:25 PM

hi @Dmitrii Egunov - are there flow runs being scheduled?

Dmitrii Egunov

04/07/2024, 9:08 AM

Yes, they are all late. There is a flow scheduled at least every 5 minutes, on top of those manually run.

Dmitrii Egunov

04/08/2024, 12:36 PM

Deleting all the deployments associated with the work pool, deleting and recreating the pool itself solved the issue. Deleting and recreating only the deployments didn’t help, it had to be work pool also. As the infra on our side still the same and started to work immediately without any changes, so I strongly believe there was a problem on Prefect Cloud side.

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.