Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

I am trying to integrate Prefect with aws services . What I understood from documentation is:
1. install prefect on aws ec2
2. write python prefect flow to trigger aws glue job

Hey <@U0497NWEMBQ>,

Prefect Cloud utilizes a hybrid execution model. What this means is that your flows will execute in your aws infrastructure and the logs will be sent to Prefect Cloud. How this works is by having a <https://docs.prefect.io/concepts/work-queues/|Prefect agent> which is intended to be a long running process that is looking for schedule work flows which are <https://docs.prefect.io/ui/deployments/|deployments> in Prefect.

For this option, you will need to install prefect and prefect-aws locally while you build your flow (python that will trigger aws glue job). Then you will create and apply a deployment that utilizes an ECSTask block to define your run infrastructure. Next you will set up a Prefect agent inside your AWS ECS. We have a nice <https://github.com/PrefectHQ/prefect-recipes/tree/main/devops/infrastructure-as-code/aws/cli-prefect2-ecs-agent|recipe> you can use to set this up.

Optionally, you can use Prefect Core where you would host a small Prefect UI (`prefect orion start` command) on your ECS and utilize that instead of Prefect Cloud.

I recommend joining the <#C048ZHT5U3U|prefect-getting-started> and <#C048K0MGHNK|prefect-aws> as well.