https://prefect.io
Join Slack
We actually got our Prefect 2 flows working in our production environment yesterday! :tada: However...
# ask-community
o
We actually got our Prefect 2 flows working in our production environment yesterday! 🎉 However, today's big daily scheduled orchestrator job has a problem I haven't seen during my 2.5 weeks of development and testing: Two of the created flows do not get picked up by the agent, they're stuck in the work queue. I just tried to start another deployment and now there's 3 in the queue. All the previous flows were picked up correctly and they all have the same tags. Has anyone run into a similar problem? I'm not sure what other information should include.. The agent is still running. There are no concurrency limits set for this work queue.
👏 1
🎉 1
🙌 1
a
Can you share how did you start your agent and how did you create your work queue? (I'm sure you did it right, just a sanity check)
can you share 
prefect version
also just a check
Since the work queue is eventually what collects all the work, you may restart the agent process and see if this way you get the agent unstuck
• mismatched prefect version or tags could be something to look at • checking logs on both sides also
you may also pause the schedule on that deployment - this will delete any currently scheduled runs, and then unpause (restart) the scheduled deployment again
it might also be a potential UI/browser issue - perhaps everything on the backend is fine? you can do hard refresh of your browser tab to be sure worst case: creating a new work queue, new agent and creating new deployments with the same configuration could be something worth looking at just throwing in a couple of things you may try
@Oscar Björhn where is your agent running?
o
Had to take a call and grab some tea, let's see now..
a
just wanted to leave 😂 will stay for a bit then
o
Haha, no worries! I'll try things out and report back, one of your suggestions will probably fix the issue.
Here are some of my answers so far: 1. Agent was started as such: New VM was created, prefect installed, PREFECT_API_URL and PREFECT_API_KEY were set and then prefect agent start "transformation-queue-prod" was run, which is the name of the queue that was already manually created in cloud. 2. Prefect agent version: 2.0.4, same as the docker images. 3. I'll try restarting the agent. It'd prefer not to, since the main orchestration flow is still "running" and waiting for these scheduled jobs to and finish, but it shouldn't be a problem.
Pausing the deployment didn't seem to remove any of the scheduled runs. Maybe because they are "late" and not "scheduled", not sure. Time to attempt an agent restart.
💯 1
a
I would try instead:
Copy code
prefect agent start -t "transformation-queue-prod"
to be explicit about the tag
this is for 2.0.4
o
Oh, the queue isn't tagged with that string though, that's actually the name of the queue. The queue is tagged with "prod" and "transformation". Is -t still the way to go?
I could use the id instead of the name, if that's preferred.
a
not to confuse you: • in 2.0.4 the tags were the matching criteria to match deployments with work queues - you would start an agent with: 
prefect agent start -t prod
• in 2.0.5 which we'll release this week, we will do it via work queue names -- the tags will still work for backward compatibility but it will become less confusing - - you would start an agent with: 
prefect agent start -q prod
- queue stands for queue and it will create a queue "prod" if it doesn't exist yet
I believe for now what will fix your issue is doing:
Copy code
prefect agent start -t prod
this will automatically create a queue prod and start an agent polling from that queue -- all deployments with tag "prod" should be picked up
o
Ah, but I need to use multiple tags.. Unless I redo my tag infrastructure a bit. 🙂
a
I'll try restarting the agent. It'd prefer not to, since the main orchestration flow is still "running" and waiting for these scheduled jobs to and finish, but it shouldn't be a problem.
you can start a new agent process without interrupting the one with active runs - you can have multiple agents polling from the same work queue
o
oh, true
a
no, you can keep all your tags - in 2.0.4 only one tag on your deployment e.g. "prod" is enough for runs from that deployment to be picked up from an agent started with 
prefect agent start -t prod
so no need to redo your tagging infra, in fact, tags remain an important concept allowing you to build meaningful categories for your deployments and runs e.g. based on projects, teams, flow owners etc.
o
Ah, right. I want this particular agent to only pick up things tagged with prod and transformation though, since we have other deployments tagged with prod and integration. Integrations are picked up by an agent on a different vm with more ram (for lifting more data), that has access to our on-prem databases. While the transformation vm/agent only runs dbt and similar commands.. If I start the agent with two tags, will it automatically listen to queues matching those two? Or is that not supported?
a
it will poll for both as you described:
LMK if not!
o
Awesome, thank you. I'll see if it helps compared to using a queue name, just being slow because I'm having some ssh issues with the vm..
🙌 1
In the end, all I did was restart the prefect agent and it picked up the 3 late jobs. Two of them failed instantly (nothing logged to server) and one is running. Since things have been running smoothly until now, I will hold off on changing how the agent is started until 2.0.5 is released. Thanks for the help by the way, really appreciate it. Will keep in mind to just try restarting the agent if it happens again.
🙌 2
🙏 2
a
nice work! thanks for the update
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by