https://prefect.io logo
Title
s

Scott Condo

04/19/2023, 2:57 PM
Hey all, I know there are several different ways to utilize prefect cloud on AWS, but I am trying to find the best solution for my companies use case. We are going to be utilizing prefect cloud for both our data transformation piece servicing many customers along with data ingestion from various api's. First question: As far as agents go, I was wondering the limitations behind agents taking in tasks. In our end objective we fear the workload agents may receive. From what I understand the relationship of agents to workpools is one to many, but workpools can only have one agent. I was wondering if you could enlighten me on some usecases you have seen in the past? Second question: Agents are used to look for deployments that need to be run that exist in a specific work queue. In this case, is there a feature or will there be in the future, for when a job needs to be run the agent will receive the flow run request and spin up infrastructure on the fly to accommodate the workload? I understand the uses of ECS, but it seems like there can be another way maybe utilizing IAM roles that have access to different resources. Third question: Do you have a specific article I can follow for getting started with prefect cloud in AWS. I have been able to launch the agent and utilize an EC2 for compute, but I want more visibility. I would thoroughly enjoy maybe some talk from people who have implemented prefect cloud utilizing EKS or ECS. I am a little lost and when I have followed the documentation I have found online it has led to no success. Thanks a bunch and I look forward to hearing back from you!
r

redsquare

04/19/2023, 3:33 PM
Hey Scott, for 3) We use EKS managed via Rancher - pretty simple using the Kubernetes Job block for infrastructure -> https://medium.com/the-prefect-blog/how-to-use-kubernetes-with-prefect-part-2-2e98cdb91c7e For ECS have you read - there is a recipes git repo to go along with the article too https://medium.com/the-prefect-blog/serverless-real-time-data-pipelines-on-aws-with-prefect-ecs-and-github-actions-1737c80da3f5
1. Agents are pretty lightweight - in k8s they just create jobs - we run lots of jobs per day with a single agent
s

Scott Condo

04/19/2023, 5:27 PM
Thanks a bunch!
is your preferred integration with eks vs ecs?
r

redsquare

04/19/2023, 5:30 PM
for me yes as it is more native/portable and avoids lock-in