Prefect Team have you seen an instance where you have worker polling a workpool, but the pool stays ...
g
Prefect Team have you seen an instance where you have worker polling a workpool, but the pool stays unhealthy and the workers don't discover the queued flows?
πŸ‘€ 1
@Jeff Hale Any advise?
j
I have not. On the work pool page, does it show that the worker recently polled the work pool?
g
yes
and the time gets updated
j
Is there a scheduled run in the pool?
g
yes
j
Is it showing as late?
g
yep
i set the worker to debug logging and it is queries the queue, but getting no flows
If i create a new queue and then connect a worker it, goes healthy
but once i schedule a flow it goes late and the queue goes unhealthy
j
What happens if you create a process work pool with no work queue and a worker to poll it?
Can you run deployments through it ok?
g
It connects, but i don't have a deployment setup to run that way
ok
i tried the same job and is failed because of setup, but that is good because the worker actually pulled it
and the queue stayed healthy
πŸ‘ 1
Tried the same with agent and it worked
So it seems to be something with the worker
So it seems to be something with the worker in my GCP hosted instance, when i run my server locally I am able to get the worker to connect and start cloudrun jobs
@Jeff Hale does the prefect worker connect differently than the agent, this seems to be only happen with the server in GCP. Is there an additional port or protocol that I need to open up[ between the server and wroker?
j
Workers are typed by infrastructure and agents are not. Workers must match the type of infrastructure of the work pool. There is a Cloud Run Work Pool. See Screenshot below of creating a Work Pool. Are you using a Cloud Run work pool type? Workers and the Cloud Run work pool type are in beta. Workers are designed to be next generation, smarter workers.
g
Correct, that is how I have it setup up
what is weird is for some reason the workers are not fined the flows
You can see they are polling, based on the dates in the screenshot
but for some reason not finding the flows
Here is a sc
Here you can see the queue with the Late runs and Last Polled timestamp
there are not errors, I turned on DEBUG Logging and it the worker s just get back a
14:33:45.650 | DEBUG   | prefect.worker.cloud-run.tester - Discovered 0 scheduled_flow_runs
πŸ€” 1
j
If you click on the workers tab, what does the Type show?
g
πŸ‘ 1
CloudRun
shouldn't I be able to use postman and find the run on the queue
because when i look at the worker code and make that api call I also get 0 runs on the queue
πŸ€” 1
j
I think we got it! I asked for a second set of eyes and my colleague found what I think is the issue. You have your concurrency limit set to 0. Change that to 10 and you should be able to have 10 flows running at once. You can change it in the UI by clicking on the three dot menu for the work pool -> edit
If you delete the 0 you should have unlimited flow run concurrency, which is the default.
g
on the workpool or the queue
Sweet got it
what is the upperbound of that?
j
Infinite - theoretically. Let’s just say a big number. πŸ™‚