Richard Alexander
03/13/2023, 12:48 PMagents
, work_pools
and storage_blocks
. I have several agents (each polling a different work pool) on the same server trying to poll jobs from the same s3
storage block, but only one of them is working. Can multiple agents/work pools be connected to the same storage block?redsquare
03/13/2023, 12:55 PMRichard Alexander
03/13/2023, 1:02 PMs3 storage block
. Can multiple agents / work pools use the same storage block?
Or, do you have ideas on how to troubleshoot why a Work Pool agent isn't working? The two that aren't working say that they are polling for work, but their jobs are sitting in a late
status, untouched by the agents that should be picking them up.redsquare
03/13/2023, 1:08 PMRichard Alexander
03/13/2023, 1:11 PMredsquare
03/13/2023, 1:12 PMRichard Alexander
03/13/2023, 1:12 PMredsquare
03/13/2023, 1:14 PMRichard Alexander
03/13/2023, 1:14 PMredsquare
03/13/2023, 1:15 PMRichard Alexander
03/13/2023, 1:21 PMwork_queue
in the deployment. It shows as default
in the UI. But that shouldn't matter, right? It should run in the default work queue.redsquare
03/13/2023, 1:41 PMRichard Alexander
03/13/2023, 1:52 PMdefault
, but still no luck. I'll try creating a work queue instead of using default.main
work queue and re-deployed without changing the agent
⢠Re-deployed with work_queue_name = 'default'
, but not changing the agent
⢠Changed the agent to explicitly poll from the default
work queue
⢠Changed the agent to explicitly poll from the main
work queue
One interesting note, when I explicitly set the agents work queue to default
the log gave some output/error messages about canceled flows that I had already deleted. But it still didn't pick up the new flow run.
Related question:
If we start an agent in a work pool, but don't specify the work queue, will it pick up work from all queues? Or no queues?
I'm not sure what else to try at this point. Any other suggestions?redsquare
03/13/2023, 2:26 PMRichard Alexander
03/13/2023, 2:27 PMredsquare
03/13/2023, 2:27 PMRichard Alexander
03/13/2023, 2:27 PMprefect agent start -p worker -q main
redsquare
03/13/2023, 2:27 PMprefect deployments ls
Richard Alexander
03/13/2023, 2:29 PMredsquare
03/13/2023, 2:29 PMRichard Alexander
03/13/2023, 2:30 PMredsquare
03/13/2023, 2:41 PMRichard Alexander
03/13/2023, 2:45 PMdefault
and the new main
queues. The main queue has 1 late run
as well since that is the last test I tried.redsquare
03/13/2023, 2:46 PMRichard Alexander
03/13/2023, 2:48 PMredsquare
03/13/2023, 2:50 PMprefect config set PREFECT_LOGGING_LEVEL=DEBUG
Richard Alexander
03/13/2023, 2:52 PMredsquare
03/13/2023, 2:53 PMRichard Alexander
03/13/2023, 2:54 PMredsquare
03/13/2023, 2:54 PMRichard Alexander
03/14/2023, 12:37 PMunhealthy
, even those that are working properly.
What should we try next?redsquare
03/14/2023, 12:41 PMRichard Alexander
03/14/2023, 12:50 PMlast-polled
column in the database somewhere that I can check?prefect agent start -p worker -q default
And get this output from the log:
Agent started! Looking for work from work pool 'worker'...
The log mentions the pool, but it doesn't say anything about the queue. Is that to be expected? Or does that indicate a problem?redsquare
03/14/2023, 2:32 PMRichard Alexander
03/14/2023, 2:34 PMredsquare
03/14/2023, 2:34 PMRichard Alexander
03/14/2023, 2:46 PMprefect server
is started on MAIN, not WORKER
I assumed that agents can be started and poll for work from the database by themselves. Is that correct or not?redsquare
03/14/2023, 2:53 PMRichard Alexander
03/14/2023, 3:15 PMPREFECT_API_DATABASE_CONNECTION_URL
. I assumed that agents poll the database directly. Is that not correct?
I know that flows have run on the WORKER server... I watched the processes get picked up with htop
and almost completely fill the available memory.redsquare
03/14/2023, 3:17 PMRichard Alexander
03/14/2023, 3:21 PMprefect server
running and connected to the same database.
Any chance that I did harm to the system?redsquare
03/14/2023, 3:23 PM