Hi. We are trying to create a base docker image with standard dependencies that we will use across multiple flows. We use ecs executor and for that we created an docker image based on the recommendation described here - https://medium.com/the-prefect-blog/the-simple-guide-to-productionizing-data-workflows-with-docker-31a5aae67c0a. Unfortunately it looks like the prefect ecs agent is not able to read the previously installed modules. We are sure that the modules are installed correctly because we see in logs that they are skipped when we try to install them using the EXTRA_PIP_PACKAGES environment package. Is there anything additionally that we have to define to make the pip installed modules available for prefect
I guess I made a wrong assumption about the ecs agent behavior. I was thinking that the flow run will be scheduled inside the container running the prefect ecs agent, but now I see that it's spawning another task and I assume that as long as it will not have the correct task definition it will always use the default prefect docker image. Am I right?
3 months ago
you're right in the sense that flow run is triggered in a separate ECS task and container rather than running directly on the agent