https://prefect.io logo
Title
b

Bernardo Galvao

01/10/2023, 2:44 PM
I am facing this error on the prefect agent :face_holding_back_tears:
All connection attempts failed
Is the Prefect agent trying to connect to
PREFECT_API_URL
?
βœ… 1
k

Kalise Richmond

01/10/2023, 4:07 PM
Hi @Bernardo Galvao, yes the agent is looking for the PREFECt_API_URL and you can provide use the --api flag on the agent to make sure that is set correctly. https://docs.prefect.io/concepts/work-queues/#agent-options
πŸ‘€ 1
βœ… 1
πŸ™Œ 1
r

Ryan Peden

01/10/2023, 5:04 PM
It should be picking it up from the
PREFECT_API_URL
setting added to your
agent
container. What version of Prefect are you running in your container? I can do a quick test with Docker Compose to see if I can help you sort this out
b

Bernardo Galvao

01/10/2023, 5:13 PM
Version 2.7.* (I can tell you the patch version soon) Mind you, this is running docker swarm that ignores
depends
keyword of compose. It restarts containers until they stop erroring out (unless I am missing something) But I can say that I set the
PREFECT_API_URL
(with
prefect config set
) in my local machine to the remote api server, to test my flows. No flows were registered in the Prefect UI, which I think is indicative that the agent is not picking up my flows (or do I need to register them as deployments first? πŸ˜…)
r

Ryan Peden

01/10/2023, 5:24 PM
You'll need to create a deployment for an agent to run your flows. You can add a schedule to your deployment, or you can runt he deployment manually via the UI It sounds like the agent is unable to connect to Orion anyway, though. There might be some extra steps to make the Docker overlay network work properly with Docker Swarm. I'll do some tests and report back. And then if needed I will update the recipe on GitHub πŸ™‚
☺️ 1
b

Bernardo Galvao

01/10/2023, 5:25 PM
alright I'll do my best to solve it too and report back πŸ™‚
r

Ryan Peden

01/10/2023, 5:27 PM
Thanks! Much appreciated πŸ™‚
b

Bernardo Galvao

01/10/2023, 5:50 PM
@Ryan Peden I rolled out @Kalise Richmond's fix in the entrypoint of the prefect agent, and it seems to be working alright πŸ™‚ The fix: (image attached) Before the fix:
$ docker logs prefect_agent.1.i4bwqmxk2zlgz3ihvj96y3smq 
Starting v2.7.6 agent connected to <http://orion:4200/api>...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from queue(s): master-q...
16:16:10.196 | ERROR   | prefect.agent - Server error '500 Internal Server Error' for url '<http://orion:4200/api/work_queues/0c44a025-39aa-42d4-b427-3096a9e2e4e8/get_runs>'
For more information check: <https://httpstatuses.com/500>
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/agent.py", line 173, in get_and_submit_flow_runs
    queue_runs = await self.client.get_runs_in_work_queue(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/orion.py", line 774, in get_runs_in_work_queue
    response = await <http://self._client.post|self._client.post>(
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 251, in send
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 749, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Server error '500 Internal Server Error' for url '<http://orion:4200/api/work_queues/0c44a025-39aa-42d4-b427-3096a9e2e4e8/get_runs>'
For more information check: <https://httpstatuses.com/500>
After the fix:
$ docker logs prefect_agent.1.u1y8r9okouus0sf9yzhw5ny7i 
Starting v2.7.6 agent connected to 172.16.8.65:4200/api...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from queue(s): master-q...
(and nothing else logged) Just one of those docker-swarm things I suppose!
r

Ryan Peden

01/10/2023, 5:56 PM
Ahh, got it. In this case if you set
PREFECT_API_URL
to http://172.16.8.65:4200/api I think it would have the same effect. But it should be possible to use the
prefect
overlay network with swarm, so I'll see if there are extra steps needed to make it work πŸ™‚ Thanks for the extra info!
πŸ‘ 1