https://prefect.io logo
Title
a

Anjaneyulu K

11/03/2022, 7:55 PM
Can anyone help
I am trying to integrate Prefect with aws services . What I understood from documentation is: 1. install prefect on aws ec2 2. write python prefect flow to trigger aws glue job
1
Is my understanding correct
k

Kalise Richmond

11/03/2022, 10:14 PM
Hey @Anjaneyulu K, Prefect Cloud utilizes a hybrid execution model. What this means is that your flows will execute in your aws infrastructure and the logs will be sent to Prefect Cloud. How this works is by having a Prefect agent which is intended to be a long running process that is looking for schedule work flows which are deployments in Prefect. For this option, you will need to install prefect and prefect-aws locally while you build your flow (python that will trigger aws glue job). Then you will create and apply a deployment that utilizes an ECSTask block to define your run infrastructure. Next you will set up a Prefect agent inside your AWS ECS. We have a nice recipe you can use to set this up. Optionally, you can use Prefect Core where you would host a small Prefect UI (
prefect orion start
command) on your ECS and utilize that instead of Prefect Cloud. I recommend joining the #prefect-getting-started and #prefect-aws as well.
👍 1
🙌 2
a

Anjaneyulu K

11/03/2022, 10:28 PM
Gotcha Thank you