Hello all, I just setup a prefect worker ECS clust...
# prefect-aws
a
Hello all, I just setup a prefect worker ECS cluster service and have been able to get very basic tasks to run successfully. However now I am testing some more complex flows and need to specify additional python packages. Its unclear how I am supposed to do this. Should I be specifying this in the deployment configuration with infra_overrides? Or do I specify this inside the ecs work pool configuration? Or is the expectation that I create a custom image that has the python environment I need and configure that image in the ecs worker pool?
k
I think the most straightforward way to do this is to build an image with your required python packages pre-installed. If you're using
prefect deploy
with a
prefect.yaml
or
.deploy
as part of a deployment script, you can specify the image you want to build when you create or update deployments.
There are a couple of ways you can get pip packages to install just before your flow run starts, but that's added time on every flow run, which isn't ideal as your list of dependencies grows.
a
yea I finally got my head wrapped around it, I am using the cloud rest API to manage my deployments over using prefect.yaml. For now I am passing "EXTRA_PIP_PACKAGES" to my infrastructure overrides but will build my own image later on and pass that to my ecs workerpool.