Jarvis Stubblefield
12/07/2022, 6:01 PMunhealthy
… I went back and ensured that my starting of the agent in production was using the correct queue. It appears to be spelled correctly and everything. However, the unhealthy
bit has never gone away and it doesn’t have a last polled date
.Kalise Richmond
12/07/2022, 6:03 PM"PREFECT_LOGGING_LEVEL": "DEBUG"
on the agent? That can help us identify more information.Jarvis Stubblefield
12/07/2022, 6:03 PMKalise Richmond
12/07/2022, 6:04 PMJarvis Stubblefield
12/07/2022, 6:04 PMKalise Richmond
12/07/2022, 6:07 PMJarvis Stubblefield
12/07/2022, 6:09 PM$ PREFECT_LOGGING_LEVEL=DEBUG prefect agent start --work-queue "tenzinga_django_prod"
Starting v2.7.0 agent connected to <https://api.prefect.cloud/api/accounts/5626ffe9-0140-4e88-babc-4a4fc614bb99/workspaces/ee8a533d-2754-420e-87f2-2d6b084984af>...
18:07:52.975 | DEBUG | prefect.client - Connecting to API at <https://api.prefect.cloud/api/accounts/5626ffe9-0140-4e88-babc-4a4fc614bb99/workspaces/ee8a533d-2754-420e-87f2-2d6b084984af/>
___ ___ ___ ___ ___ ___ _____ _ ___ ___ _ _ _____
| _ \ _ \ __| __| __/ __|_ _| /_\ / __| __| \| |_ _|
| _/ / _|| _|| _| (__ | | / _ \ (_ | _|| .` | | |
|_| |_|_\___|_| |___\___| |_| /_/ \_\___|___|_|\_| |_|
Agent started! Looking for work from queue(s): tenzinga_django_prod...
18:07:52.979 | DEBUG | prefect.agent - Checking for scheduled flow runs...
18:07:52.980 | DEBUG | prefect.agent - Checking for cancelled flow runs...
18:07:53.375 | INFO | prefect.agent - Submitting flow run '5132e016-5fc1-4422-8b4f-31cbb6f3d808'
18:07:53.941 | INFO | prefect.infrastructure.process - Opening process 'hypnotic-buffalo'...
18:07:53.942 | DEBUG | prefect.infrastructure.process - Process 'hypnotic-buffalo' running command: /home/ec2-user/.local/share/virtualenvs/ppower-spzYstDT/bin/python -m prefect.engine in /tmp/tmpl3iv1780prefect
18:07:54.072 | INFO | prefect.agent - Completed submission of flow run '5132e016-5fc1-4422-8b4f-31cbb6f3d808'
/home/ec2-user/.pyenv/versions/3.10.8/lib/python3.10/runpy.py:126: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
warn(RuntimeWarning(msg))
18:07:56.137 | DEBUG | prefect.client - Connecting to API at <https://api.prefect.cloud/api/accounts/5626ffe9-0140-4e88-babc-4a4fc614bb99/workspaces/ee8a533d-2754-420e-87f2-2d6b084984af/>
18:08:00.974 | DEBUG | prefect.agent - Checking for scheduled flow runs...
18:08:04.174 | DEBUG | prefect.agent - Checking for cancelled flow runs...
18:08:11.022 | DEBUG | prefect.agent - Checking for scheduled flow runs...
18:08:12.059 | DEBUG | prefect.agent - Checking for cancelled flow runs...
18:08:20.186 | DEBUG | prefect.agent - Checking for cancelled flow runs...
18:08:22.396 | DEBUG | prefect.agent - Checking for scheduled flow runs...
systemd
service…Kalise Richmond
12/07/2022, 6:16 PMJarvis Stubblefield
12/07/2022, 6:19 PM[Unit]
Description=prefect agent daemon
After=network.target
[Service]
PIDFile=/run/prefect_agent/pid
User=ec2-user
Group=ec2-user
RuntimeDirectory=prefect_agent
WorkingDirectory=/home/ec2-user/tenzinga/ppower
ExecStart=/home/ec2-user/.local/share/virtualenvs/ppower-spzYstDT/bin/prefect agent start --work-queue "tenzinga_django_prod"
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
Restart=always
PrivateTmp=true
[Install]
WantedBy=multi-user.target
ExecStart
to something more like this … pipenv run prefect agent start --work-queue "tenzinga_django_prod"
… that’ll ensure the environment variables are set I think…Kalise Richmond
12/07/2022, 6:23 PMsudo touch /etc/systemd/system/prefect-agent.service
echo "[Unit]
Description=Prefect Agent
After=network.target
StartLimitIntervalSec=0
[Service]
Type=simple
Restart=always
RestartSec=1
User=ec2-user
ExecStart=~/.local/bin/prefect agent start 'kr-test'
[Install]
WantedBy=multi-user.target " >> /etc/systemd/system/prefect-agent.service
Jarvis Stubblefield
12/07/2022, 6:24 PMKalise Richmond
12/07/2022, 6:24 PM/home/ec2-user/.local/bin/prefect
on the ec2 instancesystemctl status prefect-agent.service
Jarvis Stubblefield
12/07/2022, 6:27 PM/home/ec2-user/.pyenv/shims/pipenv run prefect agent start --work-queue tenzinga_django_prod
it works and starts the agent… however, now when I try to start the systemd
service and then check the status, this is what I get…
[ec2-user@ip-172-31-31-75 ppower]$ systemctl status prefect_agent.service
● prefect_agent.service - prefect agent daemon
Loaded: loaded (/etc/systemd/system/prefect_agent.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Wed 2022-12-07 18:28:20 UTC; 2s ago
Process: 30777 ExecStop=/bin/kill -s TERM $MAINPID (code=exited, status=0/SUCCESS)
Process: 31406 ExecStart=/home/ec2-user/.pyenv/shims/pipenv run prefect agent start --work-queue tenzinga_django_prod (code=exited, status=127)
Main PID: 31406 (code=exited, status=127)
Dec 07 18:28:19 ip-172-31-31-75.us-west-2.compute.internal systemd[1]: Unit prefect_agent.service entered failed state.
Dec 07 18:28:19 ip-172-31-31-75.us-west-2.compute.internal systemd[1]: prefect_agent.service failed.
Dec 07 18:28:20 ip-172-31-31-75.us-west-2.compute.internal systemd[1]: prefect_agent.service holdoff time over, scheduling restart.
Dec 07 18:28:20 ip-172-31-31-75.us-west-2.compute.internal systemd[1]: Stopped prefect agent daemon.
Dec 07 18:28:20 ip-172-31-31-75.us-west-2.compute.internal systemd[1]: start request repeated too quickly for prefect_agent.service
Dec 07 18:28:20 ip-172-31-31-75.us-west-2.compute.internal systemd[1]: Failed to start prefect agent daemon.
Dec 07 18:28:20 ip-172-31-31-75.us-west-2.compute.internal systemd[1]: Unit prefect_agent.service entered failed state.
Dec 07 18:28:20 ip-172-31-31-75.us-west-2.compute.internal systemd[1]: prefect_agent.service failed.
gunicorn
to serve the web application. It picks up the appropriate environment variables…Kalise Richmond
12/07/2022, 6:40 PMJarvis Stubblefield
12/07/2022, 6:43 PMsystemd
service still cannot find the command.ec2-user
and within the working directory
it should find the command no problem…ec2-user
and in that working directory … I know that’s not exactly how these services work, but still…/etc/systemd/system/prefect_agent.service
where I place my other service files.Kalise Richmond
12/07/2022, 6:51 PMJarvis Stubblefield
12/07/2022, 7:00 PMpipenv
which is why I thought maybe I should use the feature of pipenv run
to get that working.pyenv whence pipenv
it doesn’t list Python 3.10.8 as having pipenv
however, if I attempt to install it, it says it already exists in this version.pipenv
into the environment it was managing… then I could use the bin
directory of the virtual environment to do the following…systemd
file…
[Unit]
Description=prefect agent daemon
After=network.target
[Service]
PIDFile=/run/prefect_agent/pid
User=ec2-user
Group=ec2-user
RuntimeDirectory=prefect_agent
WorkingDirectory=/home/ec2-user/tenzinga/ppower
ExecStart=/home/ec2-user/.local/share/virtualenvs/ppower-spzYstDT/bin/pipenv run prefect agent start --work-queue "tenzinga_django_prod"
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
PrivateTmp=true
[Install]
WantedBy=multi-user.target
Restart=always
back on there…Kalise Richmond
12/07/2022, 7:28 PM