When we stop our agent, it seems that flows keep running - which is fine, but is there any way to wait for all flows to finish when stopping the agent? Would like to know that nothing is running on our agent before upgrading it.

only with containerized flows, the flow can keep running when stopping an agent because the agent deploys the flow run as a separate container process. With a local agent, when you stop your agent, the flow run will also stop and fail. to check what is scheduled to run for this agent, you can see that in the agent UI - there you can see whether the agent has any scheduled runs there is another page in the UI that shows that - "upcoming runs" in the main dashboard

Yea, it is docker containers... Hmm, need the info within a Linux shell (for an automated Jenkins deployment process, to be exact), suppose I'd have to play around with something like listing processes, listing running dockers, or a GraphQL query. I'll see what I can come up with... Very exciting for Prefect 2.0, but a bit withdrawn to interduce it within a production environment until you reach a stable release. Assuming it will be relative easy to migrate existing flows to 2.0...

Why would you want to stop the agent from CI? usually, Prefect users use CI to automatically register new flows (i.e. deploy new flows). The agent is a completely separate part of the architecture, curious to hear more about your use case

I suppose the main reason for this is some pre-prefect code, which creates its own cache files on disk. To avoid having to re-code too much, I mounted a directory from the agent into the docker when running the flow. Right now the cache is deleted by CI, which it cannot do when a flow is using it... The CI is also interacting with the agents to upload a config file (e.g. things we'd prefer not to have inside the docker image), and it uploads a systemd service file to control the agent. Last part makes it super easy to deploy new agents; clone a VM, run a new deployment, that's it.

Agents are not supposed to be used as deployment containers but more for always-on services that poll for new flow runs and deploy those. But you can always mount a custom directory to an always-on agent without having to shut this agent down - check this doc for more details https://docs.prefect.io/orchestration/agents/docker.html

Pretty much what we're doing. Using DockerRun and Module storage, with custom docker containers that have all the code installed. Agents only mount in a cache dir and config file into those dockers. The cache dir is to make it persistent across flow runs. It's okay that each agent have their own cache, for now. Hoping to get rid of the custom cache over time, as we migrate more and more code to Prefect...

This is quite an unusual setup to use ModuleStorage with DockerRun. If you leverage Docker storage instead, you can "mount" your cache_dir at registration time and you wouldn't have to mount it to the agent. If you need some examples, check those that have "docker" in the name and for Docker storage check this doc

Honestly, were a little tricky to understand what is the most fitting setup, when just starting out... Wouldn't docker storage create one container per flow though? Our docker container are quite big (~2.5GB), with almost 20 flows (and still counting) x 3 branches (production, staging, testing) it starts becoming quiet substantial, compared to our current setup. Another question on docker containers; Our flow runs seem to fail when our local dockerhub is down (500 Server Error for http+docker://localhost/v1.41/images/create?tag=[...]). I see that you can disable image pulling on the agent (--no-pull). Is there no in between options where it checks for a new image, but continues on the existing image if it for some reason can't?

Wouldn't docker storage create one container per flow though?

yes, it would to make your docker image smaller, there are many ways of approaching this. One is shown here. Which problem do you try to optimize for here specifically - saving costs on container registry or saving disk space in your execution layer? Our flow runs seem to fail when our local dockerhub is down - the best approach would be to set up some automation e.g. via flow level state handler to take some action based on that, perhaps to notify you about the failure so that you could start a new flow run once your registry is back up? Actually, your idea is quite good - you could configure that on the one hand via --no-pull and also via restart_policy on

host_config

on the Docker run: • restart_policy (dict) – • Restart the container when it exits. Configured as a dictionary with keys: ◦

Name

One of

on-failure

, or

always

. ◦

MaximumRetryCount

Number of times to restart the container on failure.

Thanks for the information, very much. Will try to have a closer look at all of it as soon as I can. Well, right now docker image size is okay, as we only have 6 images for flows. It's only if we changed to docker storage, as you suggested, I'd be a little concerned of the 3x20x2.5GB images we'd end up with. Hmm, maybe if all flows were sharing the same base image, technically only the flow code would change from storage image to storage image, not the Python dependencies... Can see I'm already using much of that Size article: • Base image: Yours (prefecthq/prefect:${prefect_version}-python3.9) to match the Prefect Server versions • Will have a look at docker-slim to see what it can do... • RUNs are already compressed • Am using multi-stage builds (mostly to save time installing the Python dependencies over and over again. • Also using docker buildkit to mount caches in (avoids having them in the image, but still retain them for the next build). • Would be interresting to look into the building with --user or virtualenv and copying the files in, instead. Our biggest issue is the number of Python dependencies that we're using .. and some of them are quite big. Doesn't help that some tasks are report generators, that need to generate diagrams and pdfs. • Also running some cleanup on the agents to remove old docker images over time. Again, thanks for the ideas!

Prefect base images are preferable over custom ones, and Prefect base images are already based on Python slim I agree that separating code dependencies (Docker image) from flow storage (e.g. S3) makes a lot of sense and can allow for building more lightweight images

Our biggest issue is the number of Python dependencies that we're using

for that, packaging those into a python package can help - one simple example