Has anyone tried using Prefect with an AWS ApplicationLoadBalancer? It keeps failing a healthtest check, regardless of what path I give it. The EC2 instance it’s pointing at is accessible and responds with a 200 to http://IP:8080 but I keep getting a failed test on it. Not saying it’s a Prefect problem, but I’m curious if anyone has hit this?

I had similar challenges with the health check, but I was running Prefect in EKS. In that case, the

PREFECT__CLOUD__AGENT__AGENT_ADDRESS

was configured to

<http://8080>

, which for some reason defaulted to only listening on

127.0.0.1

so any external requests to the endpoint would fail. To mitigate that, we set

PREFECT__CLOUD__AGENT__AGENT_ADDRESS

to

<http://0.0.0.0:8080>

to instruct it to listen on all available IP addresses configured in the Pod/EC2 instance.

Hey Scott, thanks for that! You definitely had to use the CLOUD env var rather than server? I’ll give that a try now 👍

our agent was talking to Prefect Cloud for the scheduler, if you're using the Server, I imagine the path would be different ... a quick solution would be to ensure the health endpoint is listening on an IP address which AWS can access to validate the health of the instance.

I think my problem is that I don’t actually know where the health endpoint is 🙂 I was assuming http://IP/8080 would do it, but could be wrong?

the parameter I shared was the health endpoint for the

agent

. For the server, I know there's a number of different components you could be checking the health on ... I think the

apollo

service would probably give you the best indication of overall health. When I made a helm chart for the Server component, I setup all the health checks to simply verify the TCP ports were listening. You might be able to do a health check on port 8080 for the website UI. Though I don't think they have a

/healthz

endpoint natively.

Yeah I haven’t spotted anything that’s a 1-1 for a healthhpoint, I’ll try the apollo one on the off chance the UI port is doing something funky. From what I can see in the Prefect code there’s an API healthcheck that I might be able to hit, just need to figure out which one it is 🙂

UI is open source now...build an endpoint 😉 !

You mean write actual code? Ye gods no!! 😂

Out of curiosity I took a look at the UI repo - not too bad. You might get away with a ui health check hitting

/help

. It does some unnecessary stuff for a healthcheck, but probably close to what would be needed.

I actually would if I was convinced that it was just an endpoint problem, but I’m thinking it’s related to AWS and need to figure out why I can’t get the 2 talking. Thanks for the input guys! If I get anywhere I’ll come back and let you know

FWIW, most of the time I've seen problems with healthchecks it has been due to one of two things (sorry if this is too dumbed down for you): 1. target definitions - protocols and port alignments 2. VPC and security group related issues - Is the ALB in the correct vpc, subnets picked appropriately, and does the security group on the target allow the traffic

No they’re all good points, but unfortunately I think I have them covered from my testing. The only one I’m wary of is the ports, I’ve configured everything to use 8080 but no luck. One more try with a magnifying glass, and then if I don’t get anywhere with it I’ll see if AWS can tell me what exactly is failing on their side