How do i setup the command in docker-container block to first it would run the requirements.txt file that the flow run gets from azure blob, before running the prefect engine. Following doesn't work

I would install the dependencies in your Dockerfile

FROM yourbaseimage

COPY requirements.txt .

RUN pip install -r requirements.txt

@Nate I do that, but we want a base dockerfile that can be built upon on individual flows to keep the base clean and only add the specific requirements in each container. That way we are more secure in not running into package depency issues if different projects runs different versions of same package

You could debug this locally by attempting to do a

DockerContainer(command=[…]).run()

solved the several commands problem, issue is that the requirements from blob storage does not get pulled into the docker container prior to running python -m -prefect.engine, so i can't run the command prior. Is the best practice really to have a seperate image per flow, or one standard image that all your flows use?

It depends if you’d prefer a slowdown at build time or runtime

I will bring this to the attention of the team after the weekend and see what we decide upon for now - thanks for the insight!

Since you're using

docker-container

, It sounds like the

EXTRA_PIP_PACKAGES

environment variable might help, assuming you are using one of the Prefect base images directly or building atop one? As the docs note, installing the packages at runtime adds overhead and has disadvantages, but you are the best judge of whether it makes sense for your workflow. Setting that environment variable (either during deployment or afterward via the block UI) is a bit less convenient than running pip against requirements.txt, but it might be workable? If you want to try it, you can probably flatten requirements.txt into a

pip install

compatible string you can put in the environment variable by running

awk '{gsub(" ","",$0);printf("%s ",$0)}' requirements.txt

if you're running Linux or MacOS.