Hello folks, i have a question reg Prefect cloud f...
# ask-communitya
Abhishek
09/14/2021, 12:02 PMHello folks, i have a question reg Prefect cloud flow/task execution and IP range of cloud.
Here is what my current flow does:
• single flow having multiple tasks to sync few S3 buckets. (each task runs s3.sync command via ShellTask )
• catch 22: the source S3 bucket is from different account (lets say X) and destination is bucket is in different account (lets say Y). The AWS policies are already in place to do cross account s3 sync. (the IAM role/user are already added in policy for access)
Issue:
• I need an IP address/IP range of network where this S3 task execution happens so that we can whitelist the IP range of this at source bucket account (X) policy. I am running my task on Prefect cloud - which means the request to execute sync command goes from prefect cloud network.
Is there any way i can get the IP range of prefect cloud where this execution happens ? I need to whitelist the IP range.
In case if i run prefect server on my infra then it becomes easy for me to whitelist the IP range of an AWS VPC under which the server instance runs.
But in prefect cloud i am not having clarity.
cc: @Kevin Kho
✅ 1
n
Noah Holm
09/14/2021, 12:25 PMAm I missing something here, because Prefect’s hybrid execution model means that execution would never happen in their environment. Even though you use Prefect Cloud the execution of flows must happen on your infra
📝 1
⭐ 1
a
Abhishek
09/14/2021, 12:26 PMi see. so basically prefect cloud doesn’t have compute?
n
Noah Holm
09/14/2021, 12:27 PMExactly, so for my deployment example we use Prefect Cloud which kicks off containers in AWS ECS. So the requesting IP in your case would be the IP of that ECS task
a
Abhishek
09/14/2021, 12:28 PMgot it.
do you have references so that i can look and understand/plan better for my deployment?
n
Noah Holm
09/14/2021, 12:32 PMThis article highlights the general thought behind the hybrid execution
And these two articles helped me a lot setting up our deployment this spring
Prefect on Kubernetes
Prefect on ECS Fargate
upvote 1
Noah Holm
09/14/2021, 12:33 PMYou could also keep it simpler by e.g., having a local or docker agent on a virtual server executing jobs on its own machine but you will loose a lot of the epic scaling features
a
Abhishek
09/14/2021, 1:05 PM@Noah Holm thanks a lot, this helps! 🙏
Abhishek
09/14/2021, 1:05 PMi am still learning/digesting prefect stuff :)
n
Noah Holm
09/14/2021, 1:39 PMGlad to help, good luck and enjoy! 🙌
😇 1
k
Kevin Kho
09/14/2021, 2:22 PM
Noah is exactly right, Prefect does not host compute whatsoever so you should be in control of the infrastructure and know the IP ranges
👍 1
6 Views