https://prefect.io logo
Title
j

Jessica Smith

01/20/2023, 7:35 PM
I'm working on V2 for the first time and having an issue with the agent. I have the agent set up and running in an ECS service, but the work queue still says it's never been polled and it's an unhealthy status. Shouldn't it be showing as healthy status?
z

Zanie

01/20/2023, 7:40 PM
It’s possible that your agent does not have the API URL configured correctly and is just talking to a local API instead of the remote one.
:gratitude-thank-you: 1
j

Jessica Smith

01/20/2023, 7:56 PM
i checked that, but it says that it has the proper api url and it does not say it's running in an ephemeral instance
z

Zanie

01/20/2023, 7:59 PM
Interesting! Do you have DEBUG level logs enabled for your agent?
j

Jessica Smith

01/20/2023, 8:07 PM
I do, and nothing exciting happening there. I don't have any flow deployments, would that cause this behavior? I assumed that even without deployments it would show as healthy and that it's polling
z

Zanie

01/20/2023, 8:36 PM
It should indeed! It’s possible that going to an older version means your client doesn’t actually send the healthcheck
What version is your agent running on?
j

Jessica Smith

01/20/2023, 8:40 PM
The agent is running 2.6.9
z

Zanie

01/20/2023, 8:43 PM
Hm and the agent displays “Checking for scheduled flow runs...“?
j

Jessica Smith

01/20/2023, 8:43 PM
I sent you a DM with the logs. The last log message is "Agent started! Looking for work from queue.."
z

Zanie

01/20/2023, 8:44 PM
Yeah there’s a separate debug level log when it actually polls for runs
it’d come later.
Let me get a Cloud engineer over here to look at it on the backend cc @Jake Kaplan
j

Jake Kaplan

01/20/2023, 9:04 PM
let me take a look, do you mind dming me your workspace id/work queue info?
I can see the work queue but it hasn't been polled. which tracks with no
Checking for scheduled flow runs
so I am not sure why the agent would stall there, I also don't see any requests for that work queue (which makes sense)
z

Zanie

01/20/2023, 9:16 PM
Hm the agent should display that it is checking for runs before running queries
j

Jessica Smith

01/20/2023, 9:21 PM
so i can poll the work queue from my local machine
although i still can't get the "checking for flow runs" message to appear
z

Zanie

01/20/2023, 9:21 PM
Is the ECS container able to access the outside network?
j

Jessica Smith

01/20/2023, 9:22 PM
i'm not actually sure, i'm new to ecs tasks. any advice on a simple way to check that? replace the prefect agent start command with a ping command maybe?
z

Zanie

01/20/2023, 9:24 PM
Yeah that’d be a good idea
j

Jake Kaplan

01/20/2023, 9:25 PM
prefect flow-run ls
is a pretty good proxy, as long as you have the API_URL set
j

Jessica Smith

01/20/2023, 11:52 PM
my pipeline broke and took forever to fix. so if i run the prefect flow-run ls command i get error messages from httpx
which is the same thing i get from the 2.7.* version of the prefect agent from just running prefect agent start
so i'm guessing it doesn't have external access, i'll see if i can play with that next week
Turns out the logic I had to get the api URL and key from the AWS secret was adding a beginning and end double quote, which was causing httpx to freak out
somewhere along the line it was getting encoded so it was trying to call the API at %22https...
so that's on me, sorry for the confusion. Thank you both for the help!
j

Jake Kaplan

01/24/2023, 9:48 PM
np at all! I'm glad you were able to resolve!
j

Jessica Smith

01/24/2023, 11:10 PM
FWIW, i only realized it because I tried to run a flow and that process showed an error message about not being able to connect to the API at example-url.com - it might be worthing adding the same kind of check to
prefect agent start
z

Zanie

01/25/2023, 1:02 AM
It ought to! It swallows errors for a while so that if it can’t connect temporarily it will not die. It sounds like something’s not quite right though.