Hi. We are trying to create a base docker image with standard dependencies that we will use across multiple flows. We use ecs executor and for that we created an docker image based on the recommendation described here - https://medium.com/the-prefect-blog/the-simple-guide-to-productionizing-data-workflows-with-docker-31a5aae67c0a. Unfortunately it looks like the prefect ecs agent is not able to read the previously installed modules. We are sure that the modules are installed correctly because we see in logs that they are skipped when we try to install them using the EXTRA_PIP_PACKAGES environment package. Is there anything additionally that we have to define to make the pip installed modules available for prefect
the docker image looks quite similar to what you have. The only difference is that you use a different working dir
the difference is also that we use the latest prefect image as a base
I will try your code, let's see. The devil sits usually in the details. I assume that I can test it also locally (outside of the ecs fargate runtime), am I right?
06/20/2022, 3:59 PM
it depends of what you try to test - you can build and push the image from your local machine
also you can trigger a flow run from your local machine and you can even start ECS agent from your local terminal
06/20/2022, 4:01 PM
Yes I meant to run a flow on a local docker container with the prefect ecs agent start command
Thanks. I will give it a try and will let you know if I got into the same issue
I guess I made a wrong assumption about the ecs agent behavior. I was thinking that the flow run will be scheduled inside the container running the prefect ecs agent, but now I see that it's spawning another task and I assume that as long as it will not have the correct task definition it will always use the default prefect docker image. Am I right?
06/21/2022, 11:25 AM
you're right in the sense that flow run is triggered in a separate ECS task and container rather than running directly on the agent