https://prefect.io logo
Join Slack
Powered by
I have a question regarding Work-Queues. I have m...
# ask-community
b
I have a question regarding Work-Queues. I have mine running on an EC2 Linux server, however, as soon as I kill the terminal (but not the queue) the queue seems to stop and go unhealthy. Is this common? How do I keep it running?
✅ 1
j
Work queue health is determined by whether an agent has polled that work queue within a certain amount of time. If you want to keep it showing "healthy" just keep your agent for that agent running. It's not a problem if the work queue is showing unhealthy if you don't need the agent to be polling for work.
b
Less about health. But the queue wasn’t running and my scheduled job didn’t run at night. How do I keep the agent running?
j
You can use something like supervisor or systemd to keep an agent running as a background process. See Prefect discourse for examples such as this one: https://discourse.prefect.io/t/how-to-run-a-prefect-2-agent-as-a-systemd-service-on-linux/1450 This provides restarts if the agent crashes.
b
Perfect (pun intended). Thank you!
marvin 2
Hi @Jeff Hale - sorry to bother you - I followed the article but i'm having issues running the process. For this 
ExecStart=/usr/local/bin/prefect
- how can I validate the proper path. When looking where prefect is installed it's here: 
/home/prefect/.local/lib/python3.7/site-packages
I'm a nOOb with Linux - sorry about the silly question. Any other tips i can look at?
I also validated that as the 
prefect
user I can run the prefect command and the help shows up just fin - prefect seems to be installed and working well.
Copy code
[ec2-user@ip-10-0-0-28 ~]$ sudo systemctl status prefect-agent
● prefect-agent.service - Prefect Agent
   Loaded: loaded (/etc/systemd/system/prefect-agent.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Mon 2023-01-23 21:07:17 UTC; 21min ago
  Process: 5064 ExecStart=/.local/bin/prefect agent start -q lambda-tmdb (code=exited, status=203/EXEC)
 Main PID: 5064 (code=exited, status=203/EXEC)

Jan 23 21:07:17 ip-10-0-0-28.ec2.internal systemd[1]: prefect-agent.service: main process exited, code=exited, st...EXEC
Jan 23 21:07:17 ip-10-0-0-28.ec2.internal systemd[1]: Unit prefect-agent.service entered failed state.
Jan 23 21:07:17 ip-10-0-0-28.ec2.internal systemd[1]: prefect-agent.service failed.
Jan 23 21:07:17 ip-10-0-0-28.ec2.internal systemd[1]: prefect-agent.service holdoff time over, scheduling restart.
Jan 23 21:07:17 ip-10-0-0-28.ec2.internal systemd[1]: Stopped Prefect Agent.
Jan 23 21:07:17 ip-10-0-0-28.ec2.internal systemd[1]: start request repeated too quickly for prefect-agent.service
Jan 23 21:07:17 ip-10-0-0-28.ec2.internal systemd[1]: Failed to start Prefect Agent.
Jan 23 21:07:17 ip-10-0-0-28.ec2.internal systemd[1]: Unit prefect-agent.service entered failed state.
Jan 23 21:07:17 ip-10-0-0-28.ec2.internal systemd[1]: prefect-agent.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
j
Hi Brian. Linux can be a pain.
For this 
ExecStart=/usr/local/bin/prefect
- how can I validate the proper path. When looking where prefect is installed it’s here:…
This is a little confusing - that directory in the Discourse post is created automatically for the user that was named “prefect” when that user was created . It’s not related to where the library was installed. @Ryan Peden - might be best to change the name to another user in the example. I can do it if you don’t mind.
r
I believe 
/usr/local/bin/prefect
path is where the 
prefect
executable gets installed when you run 
pip install prefect
on the system's built-in Python. It'll be a different path if you're using a venv or conda environment, but I don't think a different user account would result in a location other than 
/usr/local/bin/prefect
on its own.
I can edit the post to make it more clear; the easiest way to find the right ExecPath is by running 
which prefect
👍 1
j
With multiple users, you might also have multiple pip paths. Using pip vs pip3 might also put it in different places - if using EC2, depending on your AMI - the latest AWS Linux one has Python 2.7 and 3.7 and things can get messy.
b
Thanks @Jeff Hale and @Ryan Peden. Here is what I validated so far. First i've added paths to get Python 3 and Pip3 working as defaults for the new user created from the post called 
prefect
PIP
Copy code
$ pip --version
pip 22.3.1 from /home/prefect/.local/lib/python3.7/site-packages/pip (python 3.7)
Python
Copy code
$ python --version
Python 3.7.16
Now checking out my prefect install
Copy code
$ prefect --version
2.7.9
and
Copy code
$ which prefect
~/.local/bin/prefect
And checking out the full working director
Copy code
$ pwd
/home/prefect/.local/bin
now back to the root user
Copy code
cd /etc/systemd/system
sudo vim prefect-agent.service
Copy code
[Unit]
Description=Prefect Agent

[Service]
User=prefect
WorkingDirectory=/home/prefect
ExecStart=/.local/bin/prefect agent start -q lambda-tmdb
Restart=always

[Install]
WantedBy=multi-user.target
do you see anything wrong with that?
r
that looks good to me
b
AH!
I got it
ExecStart=/home/prefect/.local/bin/prefect agent start --work-queue "lambda-tmdb"
added 
/home/prefect/
r
ah! makes sense 😄
b
it picked up last night's 2:00 am job.
SWEET 🎉
and successfully cleared out the queued up runs 🙂
🙌 1
w
Hi, I have mine running on an Digital Ocean Ubuntu server. I follow the tutorial, but when I kill the terminal, Prefect does not trigger flow anymore. I check systemctl and prefect-agent.service is still running.
the work queue name is ' test'
but now when i run this in terminal:
Copy code
cd /etc/systemd/system
sudo vim prefect-agent.service
it shows this, do you guys think it is the problem?
Hi @Ryan Peden can you help me with that 😅
r
It looks like there's an old vim swapfile. Is there any chance you have the file open in another terminal? To be safe, I'd make a backup copy of it like so: 
cp prefect-agent.service ~/prefect-agent.service
Then, see what's in the file by running 
cat prefect-agent.service
. If it contains what you expect, delete the swapfile by running 
rm prefect-agent.service.swp
. If it looks like changes are missing, run 
vim prefect-agent.service
again and press R to recover the changes that didn't get saved last time you edited the file.
w
ok, i removed the swpfile. But it still does not work. i run:
Copy code
systemctl status prefect-agent.service
it show this. Maybe Docker is the problem?
Is there any chance you have the file open in another terminal? => no, I don't open the file in another terminal.
r
Ok, for some reason the Docker socket isn't accessible to Prefect. Does 
/var/run/docker.sock
exist? Also, are you running 
prefect-agent.service
as root? If not, you'll need to make sure the account that runs the service is part of the 
docker
group.
w
I follow this step (pic below) when start Prefect agent as systemd service. As I understand, prefect-agent.service run by Prefect account, not as root, so i also need to install Docker in Prefect account, right?
r
In this case, you'll need to add the 
prefect
user to the 
docker
user group. Depending in your Linux distro and how you installed Docker, the group might already exist. If not, you can add it by running 
sudo groupadd docker
. Then, you can add the Prefect user to the group by running 
sudo usermod -aG docker prefect
.
🙌 1
w
Thank you, let me try 😃
Thank you, it works! No more Red dot 😄
9 Views
Open in Slack
PreviousNext
Powered by