I have a question regarding Work Queues I have mine running

Question

I have a question regarding Work Queues I have mine running on an EC2 Linux server however as soon as I kill the terminal but not the queue the queue seems to stop and go unhealthy Is this common How do I keep it running

Jeff Hale · Answer

Work queue health is determined by whether an agent has polled that work queue within a certain amount of time If you want to keep it showing healthy just keep your agent for that agent running It s not a problem if the work queue is showing unhealthy if you don t need the agent to be polling for work

Brian Roepke · Answer

Less about health But the queue wasn t running and my scheduled job didn t run at night How do I keep the agent running

Jeff Hale · Answer

You can use something like supervisor or systemd to keep an agent running as a background process See Prefect discourse for examples such as this one <https discourse prefect io t how to run a prefect 2 agent as a systemd service on linux 1450|https discourse prefect io t how to run a prefect 2 agent as a systemd service on linux 1450> This provides restarts if the agent crashes

Brian Roepke · Answer

Perfect pun intended Thank you

Brian Roepke · Answer

Hi < Jeff Hale> sorry to bother you I followed the article but i m having issues running the process For this `ExecStart= usr local bin prefect` how can I validate the proper path When looking where prefect is installed it s here ` home prefect local lib python3 7 site packages`

Brian Roepke · Answer

I m a nOOb with Linux sorry about the silly question Any other tips i can look at

Brian Roepke · Answer

I also validated that as the `prefect` user I can run the prefect command and the help shows up just fin prefect seems to be installed and working well

Brian Roepke · Answer

``` ec2 user ip 10 0 0 28 ~ $ sudo systemctl status prefect agent prefect agent service Prefect Agent Loaded loaded etc systemd system prefect agent service enabled vendor preset disabled Active failed Result start limit since Mon 2023 01 23 21 07 17 UTC 21min ago Process 5064 ExecStart= local bin prefect agent start q lambda tmdb code=exited status=203 EXEC Main PID 5064 code=exited status=203 EXEC Jan 23 21 07 17 ip 10 0 0 28 ec2 internal systemd 1 prefect agent service main process exited code=exited st EXEC Jan 23 21 07 17 ip 10 0 0 28 ec2 internal systemd 1 Unit prefect agent service entered failed state Jan 23 21 07 17 ip 10 0 0 28 ec2 internal systemd 1 prefect agent service failed Jan 23 21 07 17 ip 10 0 0 28 ec2 internal systemd 1 prefect agent service holdoff time over scheduling restart Jan 23 21 07 17 ip 10 0 0 28 ec2 internal systemd 1 Stopped Prefect Agent Jan 23 21 07 17 ip 10 0 0 28 ec2 internal systemd 1 start request repeated too quickly for prefect agent service Jan 23 21 07 17 ip 10 0 0 28 ec2 internal systemd 1 Failed to start Prefect Agent Jan 23 21 07 17 ip 10 0 0 28 ec2 internal systemd 1 Unit prefect agent service entered failed state Jan 23 21 07 17 ip 10 0 0 28 ec2 internal systemd 1 prefect agent service failed Hint Some lines were ellipsized use l to show in full ```

Jeff Hale · Answer

Hi Brian Linux can be a pain gt For this `ExecStart= usr local bin prefect` how can I validate the proper path When looking where prefect is installed it s here This is a little confusing that directory in the Discourse post is created automatically for the user that was named prefect when that user was created It s not related to where the library was installed < Ryan Peden> might be best to change the name to another user in the example I can do it if you don t mind

Ryan Peden · Answer

I believe ` usr local bin prefect` path is where the `prefect` executable gets installed when you run `pip install prefect` on the system s built in Python It ll be a different path if you re using a venv or conda environment but I don t think a different user account would result in a location other than ` usr local bin prefect` on its own

Ryan Peden · Answer

I can edit the post to make it more clear the easiest way to find the right ExecPath is by running `which prefect`

Jeff Hale · Answer

With multiple users you might also have multiple pip paths Using pip vs pip3 might also put it in different places if using EC2 depending on your AMI the latest AWS Linux one has Python 2 7 and 3 7 and things can get messy

Brian Roepke · Answer

Thanks < Jeff Hale> and < Ryan Peden> Here is what I validated so far First i ve added paths to get Python 3 and Pip3 working as defaults for the new user created from the post called `prefect` PIP ```$ pip version pip 22 3 1 from home prefect local lib python3 7 site packages pip python 3 7 ``` Python ```$ python version Python 3 7 16```

Brian Roepke · Answer

Now checking out my prefect install ```$ prefect version 2 7 9``` and ```$ which prefect ~ local bin prefect```

Brian Roepke · Answer

And checking out the full working director ```$ pwd home prefect local bin```

Brian Roepke · Answer

now back to the root user ```cd etc systemd system sudo vim prefect agent service```

Brian Roepke · Answer

``` Unit Description=Prefect Agent Service User=prefect WorkingDirectory= home prefect ExecStart= local bin prefect agent start q lambda tmdb Restart=always Install WantedBy=multi user target```

Brian Roepke · Answer

do you see anything wrong with that

Ryan Peden · Answer

that looks good to me

Brian Roepke · Answer

AH

Brian Roepke · Answer

I got it

Brian Roepke · Answer

`ExecStart= home prefect local bin prefect agent start work queue lambda tmdb `

Brian Roepke · Answer

added ` home prefect `

Ryan Peden · Answer

ah makes sense smile

Brian Roepke · Answer

it picked up last night s 2 00 am job

Brian Roepke · Answer

SWEET tada

Brian Roepke · Answer

and successfully cleared out the queued up runs slightly smiling face

Will Truong · Answer

Hi I have mine running on an Digital Ocean Ubuntu server I follow the tutorial but when I kill the terminal Prefect does not trigger flow anymore I check systemctl and prefect agent service is still running

Will Truong · Answer

the work queue name is test

Will Truong · Answer

but now when i run this in terminal ```cd etc systemd system sudo vim prefect agent service``` it shows this do you guys think it is the problem

Will Truong · Answer

Hi < Ryan Peden> can you help me with that sweat smile

Ryan Peden · Answer

It looks like there s an old vim swapfile Is there any chance you have the file open in another terminal To be safe I d make a backup copy of it like so `cp prefect agent service ~ prefect agent service` Then see what s in the file by running `cat prefect agent service` If it contains what you expect delete the swapfile by running `rm prefect agent service swp` If it looks like changes are missing run `vim prefect agent service` again and press R to recover the changes that didn t get saved last time you edited the file

Will Truong · Answer

ok i removed the swpfile But it still does not work i run ```systemctl status prefect agent service``` it show this Maybe Docker is the problem

Will Truong · Answer

Is there any chance you have the file open in another terminal = gt no I don t open the file in another terminal

Ryan Peden · Answer

Ok for some reason the Docker socket isn t accessible to Prefect Does ` var run docker sock` exist Also are you running `prefect agent service` as root If not you ll need to make sure the account that runs the service is part of the `docker` group

Will Truong · Answer

I follow this step pic below when start Prefect agent as systemd service As I understand prefect agent service run by Prefect account not as root so i also need to install Docker in Prefect account right

Ryan Peden · Answer

In this case you ll need to add the `prefect` user to the `docker` user group Depending in your Linux distro and how you installed Docker the group might already exist If not you can add it by running `sudo groupadd docker` Then you can add the Prefect user to the group by running `sudo usermod aG docker prefect`

Will Truong · Answer

Thank you let me try smiley

Will Truong · Answer

Thank you it works No more Red dot smile