Hello all, I just setup a prefect worker ECS cluster service and have been able to get very basic tasks to run successfully. However now I am testing some more complex flows and need to specify additional python packages. Its unclear how I am supposed to do this. Should I be specifying this in the deployment configuration with infra_overrides? Or do I specify this inside the ecs work pool configuration? Or is the expectation that I create a custom image that has the python environment I need and configure that image in the ecs worker pool?
k
Kevin Grismore
11/16/2023, 7:20 PM
I think the most straightforward way to do this is to build an image with your required python packages pre-installed. If you're using
prefect deploy
with a
prefect.yaml
or
.deploy
as part of a deployment script, you can specify the image you want to build when you create or update deployments.
Kevin Grismore
11/16/2023, 7:22 PM
There are a couple of ways you can get pip packages to install just before your flow run starts, but that's added time on every flow run, which isn't ideal as your list of dependencies grows.
a
Alec Taggart
11/17/2023, 4:42 AM
yea I finally got my head wrapped around it, I am using the cloud rest API to manage my deployments over using prefect.yaml. For now I am passing "EXTRA_PIP_PACKAGES" to my infrastructure overrides but will build my own image later on and pass that to my ecs workerpool.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.