Hi We are trying to create a base docker image with standard

Question

Hi We are trying to create a base docker image with standard dependencies that we will use across multiple flows We use ecs executor and for that we created an docker image based on the recommendation described here <https medium com the prefect blog the simple guide to productionizing data workflows with docker 31a5aae67c0a> Unfortunately it looks like the prefect ecs agent is not able to read the previously installed modules We are sure that the modules are installed correctly because we see in logs that they are skipped when we try to install them using the EXTRA PIP PACKAGES environment package Is there anything additionally that we have to define to make the pip installed modules available for prefect

Anna Geller · Answer

You would need a custom Docker image here is an example repo with Dockerfile + aws docker build commands bash file <https github com anna geller packaging prefect flows> then you only need to reference that ECR image on your ECSRun

Roman Jendrusch · Answer

the docker image looks quite similar to what you have The only difference is that you use a different working dir

Roman Jendrusch · Answer

the difference is also that we use the latest prefect image as a base

Roman Jendrusch · Answer

I will try your code let s see The devil sits usually in the details I assume that I can test it also locally outside of the ecs fargate runtime am I right

Anna Geller · Answer

it depends of what you try to test you can build and push the image from your local machine

Anna Geller · Answer

also you can trigger a flow run from your local machine and you can even start ECS agent from your local terminal

Roman Jendrusch · Answer

Yes I meant to run a flow on a local docker container with the prefect ecs agent start command

Roman Jendrusch · Answer

Thanks I will give it a try and will let you know if I got into the same issue

Roman Jendrusch · Answer

I guess I made a wrong assumption about the ecs agent behavior I was thinking that the flow run will be scheduled inside the container running the prefect ecs agent but now I see that it s spawning another task and I assume that as long as it will not have the correct task definition it will always use the default prefect docker image Am I right

Anna Geller · Answer

you re right in the sense that flow run is triggered in a separate ECS task and container rather than running directly on the agent

Anna Geller · Answer

you can totally customize the flow run ECS task though with custom ECS task definition custom image etc docstrings here provide several examples also check this repo for more examples look for those with ECS in the name

Roman Jendrusch · Answer

Thank you very much for the quick response That helps a lot