YD
07/11/2023, 2:45 PMprefect agent start --pool default-agent-pool
and prefect agent start -q <name>
? or is there are other things I can do to understand what is going on and prevent from it happening ?Nate
07/11/2023, 2:48 PMYD
07/11/2023, 5:28 PMps aux | grep ~/tmp/prefect_agent.log &
on the cloud machine, it looks to be running
• I am using a “linode” cloud instance on the west coast. the instance does not look to be overly busy
most of the time it works fine
but when it does not, it is hard for me to tell why so that I can see how to address itNate
07/11/2023, 5:39 PMYD
07/11/2023, 7:09 PMnohup prefect agent start -q 'lin1' > ~/tmp/prefect_agent.log &
I did not see anything in the log… though will check it again, when it runs a little longer.
I rerun it, so to overwritten the logNate
07/11/2023, 7:23 PMYD
07/12/2023, 4:52 AMnohup prefect agent start -q 'lin1' --pool default-agent-pool > ~/tmp/prefect_agent.log &
I’ll see if tomorrow I’ll have less “Agent is not healthy !!!” messages
thanksNate
07/19/2023, 2:53 PMYD
07/24/2023, 3:39 AMnohup
to run prefect agent start --pool default-agent-pool --work-queue <queue name>
Nate
07/24/2023, 10:48 AMYD
07/25/2023, 6:45 AMprefect worker start --pool YOUR_WORK_POOL_NAME
with
prefect agent start --pool default-agent-pool --work-queue lin1
Nate
07/25/2023, 4:02 PMprefect.yaml
and define appropriate pull
steps for them. the pull
step tells the worker where / how to pull flow code. otherwise it gets its job config from the work pool, which you attach to a deployment when you create it like. workers are just strongly typed (only work with one type of workpool) agents that can prepare for flow runs via a pull
step in whatever way you need (shell script, grab env vars etc)YD
07/25/2023, 4:07 PMprefect-worker.service
be OK ?
[Unit]
Description=Prefect Worker
[Service]
User=prefect
WorkingDirectory=/home/prefect
ExecStart=/usr/local/bin/prefect agent start --pool default-agent-pool --work-queue lin1
Restart=always
[Install]
WantedBy=multi-user.target
nohup
method I use to keep the agent runningNate
07/25/2023, 4:09 PMYD
07/25/2023, 4:10 PMNate
07/25/2023, 4:11 PMYD
07/25/2023, 10:44 PMNate
07/25/2023, 10:44 PMYD
07/25/2023, 10:45 PMprefect agent start --pool default-agent-pool --work-queue lin1
it works wellNate
07/25/2023, 10:45 PMprefect
user?YD
07/25/2023, 10:46 PMsudo systemctl daemon-reload
sudo systemctl enable prefect-worker
sudo systemctl start prefect-worker
it looked like everything works fineNate
07/25/2023, 10:47 PMjournalctl -u prefect-worker
YD
07/25/2023, 10:48 PM$ ps aux | grep prefect
prefect 3293262 143 0.6 135440 109988 ? Rs 22:47 0:01 /usr/bin/python3 /usr/local/bin/prefect agent start --pool default-agent-pool --work-queue lin1
youval 3293265 0.0 0.0 6244 704 pts/1 R+ 22:47 0:00 grep prefect
Checking log$ journalctl -u prefect-worker
Hint: You are currently not seeing messages from other users and the system.
Users in groups 'adm', 'systemd-journal' can see all messages.
Pass -q to turn off this notice.
-- Journal begins at Mon 2023-03-06 05:46:28 UTC, ends at Tue 2023-07-25 22:47:46 UTC. --
-- No entries --
Nate
07/25/2023, 10:50 PMprefect
userYD
07/25/2023, 10:51 PMNate
07/25/2023, 10:53 PMYD
07/25/2023, 10:54 PM$ cat /etc/systemd/system/prefect-worker.service
[Unit]
Description=Prefect Worker
[Service]
User=prefect
WorkingDirectory=/home/prefect
ExecStart=/usr/local/bin/prefect agent start --pool default-agent-pool --work-queue lin1
Restart=always
[Install]
WantedBy=multi-user.target
Prefect Version:
$ prefect --version
2.7.7
Nate
07/25/2023, 10:54 PMYD
07/25/2023, 10:55 PMjournalctl -u prefect-worker
that looks at the tail of the log ?Nate
07/25/2023, 11:01 PMjournalctl -u prefect-worker
that looks at the tail of the log ?Marvin
07/25/2023, 11:01 PMjournalctl
, you can use the -f
or --follow
option. In your case, to follow the logs of prefect-worker
, you'd use the command journalctl -u prefect-worker -f
. This will keep the output open and print new entries as they come in, much like the behavior of tail -f
. Please try not to enjoy this knowledge too much, wouldn't want you to get your hopes up.Nate
07/25/2023, 11:01 PMMarvin
07/25/2023, 11:02 PMjournalctl
, you can use the -n
or --lines
option, followed by the number of lines you want to view.
So, to see the last N lines of logs for prefect-worker
, you would use the command journalctl -u prefect-worker -n N
where 'N' is the number of lines you want to see. For instance, journalctl -u prefect-worker -n 20
will show you the last 20 lines.
Just remember: no matter how many lines you read, it will never fill the void. But go ahead, dig into those logs.YD
07/25/2023, 11:04 PMjournalctl -u prefect-worker -n 50 -e
ModuleNotFoundError("No module named 'data_collection'")
for flows that run for a long timeNate
07/25/2023, 11:25 PMYD
07/25/2023, 11:25 PMpip3 list | grep data
in my user has the package, but “prefect” systemd user does notYD
07/25/2023, 11:27 PMNate
07/25/2023, 11:29 PMExecStart=/usr/local/bin/
with ExecStart=/usr/your/interpreter/with/deps/
or yeah, you can install docker on the VM, add prefect
to that user group and let it submit flow runs as containersYD
07/25/2023, 11:30 PMNate
07/25/2023, 11:32 PMYD
07/25/2023, 11:47 PM/usr/bin/python3
I should try
ExecStart=/usr/bin/prefect agent start --pool default-agent-pool --work-queue lin1
in the systemd
?Nate
07/25/2023, 11:51 PM/usr/bin/python3
then yes, but you could also have a venv that has prefect and your deps and use that
❯ mamba activate prefect-2
❯ where prefect
/Users/nate/micromamba/envs/prefect-2/bin/prefect
❯ mamba deactivate
❯ mamba activate bleeding-prefect
❯ where prefect
/Users/nate/micromamba/envs/bleeding-prefect/bin/prefect
so if I was running linux and wanted to use my prefect-2
interpreter to run the agent process i'd do
ExecStart=/Users/nate/micromamba/envs/prefect-2/bin/prefect agent start --pool default-agent-pool --work-queue lin1
where here, micromamba is used for venv management, but you could just use a venv you create like
python3 -m venv myvenv
install prefect and then do where prefect
to get the path to the interpreterYD
07/25/2023, 11:56 PMWorkingDirectory=/home/prefect
in the prefect-worker.service
file ?Nate
07/25/2023, 11:57 PMYD
07/26/2023, 12:13 AMExecStart=/home/youval/.local/bin/prefect agent start --pool default-agent-pool --work-queue lin1
the agent is running but still does not recognize the pythonWorkingDirectory=/home/youval
nohup
that is not stable but it does work …Nate
07/26/2023, 12:23 AMYD
07/26/2023, 12:24 AMNate
07/26/2023, 12:30 AMnohup
isn't designed to revive itself, whereas systemd is, so if you were able to get that working, that would be ideal
let me know if i can help with something else!YD
07/26/2023, 12:30 AMNate
07/26/2023, 12:30 AMYD
07/26/2023, 12:31 AMNate
07/26/2023, 12:31 AMYD
07/26/2023, 2:30 PMJul 26 14:27:02 localhost prefect[3294881]: request
Jul 26 14:27:02 localhost prefect[3294881]: return await self.send(request, auth=auth,
Jul 26 14:27:02 localhost prefect[3294881]: follow_redirects=follow_redirects)
Jul 26 14:27:02 localhost prefect[3294881]: File "/home/youval/.local/lib/python3.9/site-packages/prefect/client/base.py",
Jul 26 14:27:02 localhost prefect[3294881]: line 251, in send
Jul 26 14:27:02 localhost prefect[3294881]: response = await self._send_with_retry(
Jul 26 14:27:02 localhost prefect[3294881]: File "/home/youval/.local/lib/python3.9/site-packages/prefect/client/base.py",
Jul 26 14:27:02 localhost prefect[3294881]: line 193, in _send_with_retry
Jul 26 14:27:02 localhost prefect[3294881]: response = await request()
Jul 26 14:27:02 localhost prefect[3294881]: File "/usr/local/lib/python3.9/dist-packages/httpx/_client.py", line 1620, in
Jul 26 14:27:02 localhost prefect[3294881]: send
Jul 26 14:27:02 localhost prefect[3294881]: response = await self._send_handling_auth(
Jul 26 14:27:02 localhost prefect[3294881]: File "/usr/local/lib/python3.9/dist-packages/httpx/_client.py", line 1648, in
Jul 26 14:27:02 localhost prefect[3294881]: _send_handling_auth
Jul 26 14:27:02 localhost prefect[3294881]: response = await self._send_handling_redirects(
Jul 26 14:27:02 localhost prefect[3294881]: File "/usr/local/lib/python3.9/dist-packages/httpx/_client.py", line 1685, in
Jul 26 14:27:02 localhost prefect[3294881]: _send_handling_redirects
Jul 26 14:27:02 localhost prefect[3294881]: response = await self._send_single_request(request)
Jul 26 14:27:02 localhost prefect[3294881]: File "/usr/local/lib/python3.9/dist-packages/httpx/_client.py", line 1722, in
Jul 26 14:27:02 localhost prefect[3294881]: _send_single_request
Jul 26 14:27:02 localhost prefect[3294881]: response = await transport.handle_async_request(request)
Jul 26 14:27:02 localhost prefect[3294881]: File "/usr/local/lib/python3.9/dist-packages/httpx/_transports/default.py",
Jul 26 14:27:02 localhost prefect[3294881]: line 353, in handle_async_request
Jul 26 14:27:02 localhost prefect[3294881]: resp = await self._pool.handle_async_request(req)
Jul 26 14:27:02 localhost prefect[3294881]: File "/usr/lib/python3.9/contextlib.py", line 135, in __exit__
Jul 26 14:27:02 localhost prefect[3294881]: self.gen.throw(type, value, traceback)
Jul 26 14:27:02 localhost prefect[3294881]: File "/usr/local/lib/python3.9/dist-packages/httpx/_transports/default.py",
Jul 26 14:27:02 localhost prefect[3294881]: line 77, in map_httpcore_exceptions
Jul 26 14:27:02 localhost prefect[3294881]: raise mapped_exc(message) from exc
Jul 26 14:27:02 localhost prefect[3294881]: httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state
Jul 26 14:27:02 localhost prefect[3294881]: ConnectionState.CLOSED
Jul 26 14:27:02 localhost prefect[3294881]: Backing off due to consecutive errors, using increased interval of 60.0s.
Nate
07/26/2023, 2:35 PMpip install -U httpcore
otherwise you can disable HTTP2 if you're not able to update to a worker
export PREFECT_API_ENABLE_HTTP2=false
YD
07/27/2023, 12:02 AMhttpcore
will see what will happen tomorrow