Hello all! We're trying to add auto-scaling rules for the prefect agents (e.g. local agent) Is it okay to add/remove prefect agents dynamically? I can see on the Prefect Cloud UI that agents are not removed from the dashboard after stopping. Also, is it a valid use case for Prefect to run a flow task to spin-up another agent and run a subflow on that agent?

@Bogdan Bliznyuk while it could be possible in theory, there are better ways of scaling out. We do have special agents built to run on clusters of compute such as KubernetesAgent, ECSAgent and VertexAgent. Local agent and Docker agent are fully capable of locally executing flows in conjunction with the Prefect API, but they are intended to be used for workloads that can run on a single host. Agents in general are lightweight processes that are intended to run 24/7, so treating them as ephemeral compute can cause unintended side effects. What seems to be fitting well in your use case is a Kubernetes cluster with autoscaling for compute nodes. You can check out this thread discussing the same topic.

Thank you very much for the response! We were evaluating ECS Agent - it has limitations on the persistent storage and other resources that doesn't fit our needs. We came up with this architecture that (as we hoped) would allow us to easily scale flow executions without the need of the complex k8s infrastructure:

ECS supports persistent storage through EFS across all capacity providers, even Fargate. If managing a Kubernetes cluster is too complex for you, you can use a fully managed and even a fully serverless option provided by AWS EKS. Most of the complexity is abstracted away from you if you combine that with Prefect.

Thank you! We've evaluated EFS integration, but our cases require quite intensive disk usage, so we'd like to have something SSD based

Are you talking about ECS or ECS Fargate?

We were evaluating fargate, but I understand now it also makes sense to look at ECS

this may be helpful for your architecture decisions https://docs.prefect.io/orchestration/#architecture-overview

Hey! Another random question: is it possible to specify in the flow's DockerRun the

base_url

parameter? https://docker-py.readthedocs.io/en/stable/client.html#docker.client.DockerClient

Yes, this should be possible through the host_config dictionary:

DockerRun(host_config=dict(base_url="your_url"))

Great stuff! Thank you very much!