Is there a healthcheck route we can use for an

agent

/

executor

docker service? I saw a feb 5 PR around some storage healthchecks, but not seeing docs for instrumented monitoring here. Ideally something curl-able, like,

healthcheck:
      test: ["CMD-SHELL", "curl -sSf <http://prefect/health> | jq .code | grep 200 || exit 1"]

Hey Leo! Which agent are you running and where? The k8s agent has a health check route but none of the other ones do at the moment. Is the goal to trigger a restart in the event the health check fails?

we're trying to figure out secure (federated) & observable & auto-restarting executors. basic soln we came to is docker-compose executors put onto every server donated to us. the toy setup is used a prefect container base, but we'll be switching to a nvidia rapids base image and pip/conda installing prefect on top (we're trying to reuse gpu/pydata/etc. deps across all envs, so won't use prefect base). we're using standard docker monitoring etc. stacks, so hoping for some in-inprocess rest endpoint we can just poll on.

when you’re talking about “executor health” what are you referring to? The only executor type that is long-lasting is a Dask executor but you don’t appear to be doing anything with Dask

We have a growing number of compute servers. Each one runs a docker container with

prefect

installed, and afaict,

prefect

will poll the central server for new tasks. We can make sure the docker container itself stays running, but not sure how to tell if

prefect

gets wedged .

It sounds like you’re running multiple Prefect agents on various servers; what type of agents are you running? Local Agents?

Not sure. The prototype has

RUN prefect backend server

. Ultimately these will be

<http://rapids.ai|rapids.ai>

tasks + neural network stuff. We're not using prefect for its dask capabilities, just task dispatch & reporting.

prefect backend server

doesn’t really do anything, it just updates your local user configuration to point to

localhost:4200

for the API instead of Cloud

Ah I see our entrypoint.sh also has:

# Keep the container running
prefect agent start

got it, yea that’s a Local Agent then 👍

But we can switch to whatever other agent, we're trying to figure out the right way to do it

yea, local agent is perfectly fine — the local agent will submit flows to run in subprocesses. Generally we recommend using

supervisord

to manage the parent process running the agent

yup exactly

Yeah supervisord has a healthcheck framework, but it still comes down to the prefect agent process having some way of doing a "oh hi, yep just checked, I am indeed OK for what I consider OK to mean"

we could definitely look into that as an enhancement though!

Yep!

awesome thank you Leo!

local agent

, right?

yup yup

https://github.com/PrefectHQ/prefect/issues/2313 There are fancier forms of this, so just wrote the bog simple & standard one :)