Thread
#prefect-community
    Roman Jendrusch

    Roman Jendrusch

    3 months ago
    Hi. We are trying to create a base docker image with standard dependencies that we will use across multiple flows. We use ecs executor and for that we created an docker image based on the recommendation described here - https://medium.com/the-prefect-blog/the-simple-guide-to-productionizing-data-workflows-with-docker-31a5aae67c0a. Unfortunately it looks like the prefect ecs agent is not able to read the previously installed modules. We are sure that the modules are installed correctly because we see in logs that they are skipped when we try to install them using the EXTRA_PIP_PACKAGES environment package. Is there anything additionally that we have to define to make the pip installed modules available for prefect
    Anna Geller

    Anna Geller

    3 months ago
    You would need a custom Docker image - here is an example repo with Dockerfile + aws_docker_build_commands.bash file https://github.com/anna-geller/packaging-prefect-flows then you only need to reference that ECR image on your ECSRun
    Roman Jendrusch

    Roman Jendrusch

    3 months ago
    the docker image looks quite similar to what you have. The only difference is that you use a different working dir
    the difference is also that we use the latest prefect image as a base
    I will try your code, let's see. The devil sits usually in the details. I assume that I can test it also locally (outside of the ecs fargate runtime), am I right?
    Anna Geller

    Anna Geller

    3 months ago
    it depends of what you try to test - you can build and push the image from your local machine
    also you can trigger a flow run from your local machine and you can even start ECS agent from your local terminal
    Roman Jendrusch

    Roman Jendrusch

    3 months ago
    Yes I meant to run a flow on a local docker container with the prefect ecs agent start command
    Thanks. I will give it a try and will let you know if I got into the same issue
    I guess I made a wrong assumption about the ecs agent behavior. I was thinking that the flow run will be scheduled inside the container running the prefect ecs agent, but now I see that it's spawning another task and I assume that as long as it will not have the correct task definition it will always use the default prefect docker image. Am I right?
    Anna Geller

    Anna Geller

    3 months ago
    you're right in the sense that flow run is triggered in a separate ECS task and container rather than running directly on the agent
    you can totally customize the flow run ECS task though with custom ECS task definition, custom image etc - docstrings here provide several examples https://docs.prefect.io/api/latest/run_configs.html#ecsrun also check this repo for more examples - look for those with ECS in the name https://github.com/anna-geller/packaging-prefect-flows/
    Roman Jendrusch

    Roman Jendrusch

    3 months ago
    Thank you very much for the quick response. That helps a lot