Setting up an EC2 agent service

Question

So I noticed this last night as I was getting everything setup but it would seem that my Work Queue is `unhealthy` I went back and ensured that my starting of the agent in production was using the correct queue It appears to be spelled correctly and everything However the `unhealthy` bit has never gone away and it doesn t have a `last polled date`

Jarvis Stubblefield · Answer

I now have a flow that is 3 hours late in running because nothing picked it up I m working to resolve Currently in production the prefect API values are done through environment variables and not configured in the CLI

Kalise Richmond · Answer

Do you have logs from the agent you can share Also have you set ` PREFECT LOGGING LEVEL DEBUG ` on the agent That can help us identify more information

Jarvis Stubblefield · Answer

I did create a profile on the CLI which just has a name and no values Could it be looking at the profile and then not properly connecting since it doesn t have the API Key Workspace URL

Jarvis Stubblefield · Answer

I ll pop into the server and restart the agent

Kalise Richmond · Answer

It definitely could be that The unhealthy means that there is no connection between the agent and the workqueue which explains the Late flows

Jarvis Stubblefield · Answer

So I can set the logs

Jarvis Stubblefield · Answer

I made the profile so it would stop complaining that it didn t exist

Jarvis Stubblefield · Answer

lol

Jarvis Stubblefield · Answer

I can populate it with the appropriate values but was just trying to let the ENV handle that

Kalise Richmond · Answer

Are you trying to use the environment variables like this <https docs prefect io concepts settings overriding defaults with environment variables>

Kalise Richmond · Answer

Oh I just found a new doc I ve never read that might be helpful Looks like there can be conflicts with the profile and environment variables <https docs prefect io concepts settings conflicts with environment variables>

Jarvis Stubblefield · Answer

Seems so I have proven this by going ahead and running the agent directly within the environment and it worked Running as a Systemd service is where the issue is

Jarvis Stubblefield · Answer

It just picked up the late run and ran it when I did the following ```$ PREFECT LOGGING LEVEL=DEBUG prefect agent start work queue tenzinga django prod Starting v2 7 0 agent connected to 18 07 52 975 | DEBUG | prefect client Connecting to API at | | | | | | | | | | | || || | | | | || ` | | | | | | | | | | | | | | | | | | | Agent started Looking for work from queue s tenzinga django prod 18 07 52 979 | DEBUG | prefect agent Checking for scheduled flow runs 18 07 52 980 | DEBUG | prefect agent Checking for cancelled flow runs 18 07 53 375 | INFO | prefect agent Submitting flow run 5132e016 5fc1 4422 8b4f 31cbb6f3d808 18 07 53 941 | INFO | prefect infrastructure process Opening process hypnotic buffalo 18 07 53 942 | DEBUG | prefect infrastructure process Process hypnotic buffalo running command home ec2 user local share virtualenvs ppower spzYstDT bin python m prefect engine in tmp tmpl3iv1780prefect 18 07 54 072 | INFO | prefect agent Completed submission of flow run 5132e016 5fc1 4422 8b4f 31cbb6f3d808 home ec2 user pyenv versions 3 10 8 lib python3 10 runpy py 126 RuntimeWarning prefect engine found in sys modules after import of package prefect but prior to execution of prefect engine this may result in unpredictable behaviour warn RuntimeWarning msg 18 07 56 137 | DEBUG | prefect client Connecting to API at 18 08 00 974 | DEBUG | prefect agent Checking for scheduled flow runs 18 08 04 174 | DEBUG | prefect agent Checking for cancelled flow runs 18 08 11 022 | DEBUG | prefect agent Checking for scheduled flow runs 18 08 12 059 | DEBUG | prefect agent Checking for cancelled flow runs 18 08 20 186 | DEBUG | prefect agent Checking for cancelled flow runs 18 08 22 396 | DEBUG | prefect agent Checking for scheduled flow runs ```

Jarvis Stubblefield · Answer

The flow run happens a bit below that but it is working when executed manually in the environment

Jarvis Stubblefield · Answer

< Kalise Richmond> I do not explicitly set the profile in the `systemd` service

Jarvis Stubblefield · Answer

So that collision shouldn t be the issue

Jarvis Stubblefield · Answer

It would seem to me that maybe be it s not picking up the correct environment setup

Jarvis Stubblefield · Answer

The runtime warning in the above output was odd and I m not sure why it is complaining

Kalise Richmond · Answer

How do you define the systemd service I found this discourse that Ryan wrote with an example <https discourse prefect io t how to run a prefect 2 agent as a systemd service on linux 1450>

Jarvis Stubblefield · Answer

Here is my example ``` Unit Description=prefect agent daemon After=network target Service PIDFile= run prefect agent pid User=ec2 user Group=ec2 user RuntimeDirectory=prefect agent WorkingDirectory= home ec2 user tenzinga ppower ExecStart= home ec2 user local share virtualenvs ppower spzYstDT bin prefect agent start work queue tenzinga django prod ExecReload= bin kill s HUP $MAINPID ExecStop= bin kill s TERM $MAINPID Restart=always PrivateTmp=true Install WantedBy=multi user target```

Jarvis Stubblefield · Answer

Though I m thinking I might need to change the `ExecStart` to something more like this `pipenv run prefect agent start work queue tenzinga django prod ` that ll ensure the environment variables are set I think

Kalise Richmond · Answer

Ya I m just looking at an example I wrote a long time ago and this is what I had ```sudo touch etc systemd system prefect agent service echo Unit Description=Prefect Agent After=network target StartLimitIntervalSec=0 Service Type=simple Restart=always RestartSec=1 User=ec2 user ExecStart=~ local bin prefect agent start kr test Install WantedBy=multi user target gt gt etc systemd system prefect agent service```

Jarvis Stubblefield · Answer

So I need a path to pipenv let me get that in the right spot wink

Kalise Richmond · Answer

I think I had prefect installed at ` home ec2 user local bin prefect` on the ec2 instance

Kalise Richmond · Answer

are you able to see anything about the status or logs from systemd when you start the daemon `systemctl status prefect agent service`

Jarvis Stubblefield · Answer

Yea right now it s complaining about my pathing not being absolute

Jarvis Stubblefield · Answer

Okay when I run the command ` home ec2 user pyenv shims pipenv run prefect agent start work queue tenzinga django prod` it works and starts the agent however now when I try to start the `systemd` service and then check the status this is what I get ``` ec2 user ip 172 31 31 75 ppower $ systemctl status prefect agent service prefect agent service prefect agent daemon Loaded loaded etc systemd system prefect agent service enabled vendor preset disabled Active failed Result start limit since Wed 2022 12 07 18 28 20 UTC 2s ago Process 30777 ExecStop= bin kill s TERM $MAINPID code=exited status=0 SUCCESS Process 31406 ExecStart= home ec2 user pyenv shims pipenv run prefect agent start work queue tenzinga django prod code=exited status=127 Main PID 31406 code=exited status=127 Dec 07 18 28 19 ip 172 31 31 75 us west 2 compute internal systemd 1 Unit prefect agent service entered failed state Dec 07 18 28 19 ip 172 31 31 75 us west 2 compute internal systemd 1 prefect agent service failed Dec 07 18 28 20 ip 172 31 31 75 us west 2 compute internal systemd 1 prefect agent service holdoff time over scheduling restart Dec 07 18 28 20 ip 172 31 31 75 us west 2 compute internal systemd 1 Stopped prefect agent daemon Dec 07 18 28 20 ip 172 31 31 75 us west 2 compute internal systemd 1 start request repeated too quickly for prefect agent service Dec 07 18 28 20 ip 172 31 31 75 us west 2 compute internal systemd 1 Failed to start prefect agent daemon Dec 07 18 28 20 ip 172 31 31 75 us west 2 compute internal systemd 1 Unit prefect agent service entered failed state Dec 07 18 28 20 ip 172 31 31 75 us west 2 compute internal systemd 1 prefect agent service failed ```

Jarvis Stubblefield · Answer

Oddly enough I use a command similar to my first posting to start `gunicorn` to serve the web application It picks up the appropriate environment variables

Kalise Richmond · Answer

so weird Ok so at least we know that the path is correct for starting the ExecStart command Now to figure out why the systemd is failing

Kalise Richmond · Answer

oh where did you store the service It seems like its still exiting because it cannot find the command

Jarvis Stubblefield · Answer

Yes the `systemd` service still cannot find the command

Jarvis Stubblefield · Answer

If it is executing as `ec2 user` and within the `working directory` it should find the command no problem

Jarvis Stubblefield · Answer

As I am logged in as `ec2 user` and in that working directory I know that s not exactly how these services work but still

Jarvis Stubblefield · Answer

The service is saved in ` etc systemd system prefect agent service` where I place my other service files

Jarvis Stubblefield · Answer

I m not certain that it would pick up the appropriate values from the config but maybe I just need to set the config values and call it I would prefer to get it working from the environment

Kalise Richmond · Answer

I think you should be able to use the environment variables no problem From the place you saved the service can you run the ExecStart command manually and does that start the agent

Jarvis Stubblefield · Answer

Both the original command above and the command I changed it to run the agent and connect to Prefect Cloud

Jarvis Stubblefield · Answer

However the systemd service does not disappointed

Jarvis Stubblefield · Answer

If I change it back to the original pathing it doesn t error it runs just not connecting to the cloud as those environment variables are within the virtualenv setup with `pipenv` which is why I thought maybe I should use the feature of `pipenv run` to get that working

Jarvis Stubblefield · Answer

Okay this might be part of it

Jarvis Stubblefield · Answer

I have python set to 3 10 8 within my project directory

Jarvis Stubblefield · Answer

However I don t believe I have pipenv installed in that version of python

Jarvis Stubblefield · Answer

Well it says I do

Jarvis Stubblefield · Answer

When I run `pyenv whence pipenv` it doesn t list Python 3 10 8 as having `pipenv` however if I attempt to install it it says it already exists in this version

Jarvis Stubblefield · Answer

Okay so I THINK the solution was

Jarvis Stubblefield · Answer

I installed `pipenv` into the environment it was managing then I could use the `bin` directory of the virtual environment to do the following

Jarvis Stubblefield · Answer

Working `systemd` file ``` Unit Description=prefect agent daemon After=network target Service PIDFile= run prefect agent pid User=ec2 user Group=ec2 user RuntimeDirectory=prefect agent WorkingDirectory= home ec2 user tenzinga ppower ExecStart= home ec2 user local share virtualenvs ppower spzYstDT bin pipenv run prefect agent start work queue tenzinga django prod ExecReload= bin kill s HUP $MAINPID ExecStop= bin kill s TERM $MAINPID PrivateTmp=true Install WantedBy=multi user target```

Jarvis Stubblefield · Answer

Adding `Restart=always` back on there

Jarvis Stubblefield · Answer

Well < Kalise Richmond> thanks for being my rubber ducky and providing input and suggestions smile

Jarvis Stubblefield · Answer

Seems it is up and running the queue is showing healthy excited to see runs happen automatically

Jarvis Stubblefield · Answer

Tomorrow morning will likely be the first

Jarvis Stubblefield · Answer

I think I need to adjust the timing for the schedule as it is running at 8 45 in the morning instead of 14 45 in the afternoon oddly enough that is my timezone difference which I didn t take into account

Kalise Richmond · Answer

Yay I m so glad you got this working marvin