Hello all! We're trying to add auto-scaling rules ...
# prefect-server
b
Hello all! We're trying to add auto-scaling rules for the prefect agents (e.g. local agent) Is it okay to add/remove prefect agents dynamically? I can see on the Prefect Cloud UI that agents are not removed from the dashboard after stopping. Also, is it a valid use case for Prefect to run a flow task to spin-up another agent and run a subflow on that agent?
a
@Bogdan Bliznyuk while it could be possible in theory, there are better ways of scaling out. We do have special agents built to run on clusters of compute such as KubernetesAgent, ECSAgent and VertexAgent. Local agent and Docker agent are fully capable of locally executing flows in conjunction with the Prefect API, but they are intended to be used for workloads that can run on a single host. Agents in general are lightweight processes that are intended to run 24/7, so treating them as ephemeral compute can cause unintended side effects. What seems to be fitting well in your use case is a Kubernetes cluster with autoscaling for compute nodes. You can check out this thread discussing the same topic.
b
Thank you very much for the response! We were evaluating ECS Agent - it has limitations on the persistent storage and other resources that doesn't fit our needs. We came up with this architecture that (as we hoped) would allow us to easily scale flow executions without the need of the complex k8s infrastructure:
But, as per your input I see it's not a good use-case for Prefect Agents
a
ECS supports persistent storage through EFS across all capacity providers, even Fargate. If managing a Kubernetes cluster is too complex for you, you can use a fully managed and even a fully serverless option provided by AWS EKS. Most of the complexity is abstracted away from you if you combine that with Prefect.
b
Thank you! We've evaluated EFS integration, but our cases require quite intensive disk usage, so we'd like to have something SSD based
+ it is still limited on the CPU/memory resources with fargate
a
Are you talking about ECS or ECS Fargate?
ECS with EC2 capacity provider has only limits that EC2 instances you chose have. You can have as much CPU and memory as you wish
b
We were evaluating fargate, but I understand now it also makes sense to look at ECS
yeah, makes sense
thank you!
a
this may be helpful for your architecture decisions https://docs.prefect.io/orchestration/#architecture-overview
🙌 1
b
Hey! Another random question: is it possible to specify in the flow's DockerRun the
base_url
parameter? https://docker-py.readthedocs.io/en/stable/client.html#docker.client.DockerClient
a
Yes, this should be possible through the host_config dictionary:
Copy code
DockerRun(host_config=dict(base_url="your_url"))
b
Great stuff! Thank you very much!