https://prefect.io logo
Title
n

Nic

09/23/2022, 5:35 PM
How do i setup the command in docker-container block to first it would run the requirements.txt file that the flow run gets from azure blob, before running the prefect engine. Following doesn't work
n

Nate

09/23/2022, 5:38 PM
I would install the dependencies in your Dockerfile
FROM yourbaseimage

COPY requirements.txt .

RUN pip install -r requirements.txt
n

Nic

09/23/2022, 5:40 PM
@Nate I do that, but we want a base dockerfile that can be built upon on individual flows to keep the base clean and only add the specific requirements in each container. That way we are more secure in not running into package depency issues if different projects runs different versions of same package
The flow is then grabbed from the blob storage, keeping flow and container independent of each other. So we're looking for a way to do this
z

Zanie

09/23/2022, 6:03 PM
You could debug this locally by attempting to do a
DockerContainer(command=[…]).run()
👍 1
To run multiple commands, you’ll need to do a bit more
👍 1
1
e.g.
DockerContainer(command=["bash", "-c", "pip install cowsay && echo hi!"])
👍 1
n

Nic

09/23/2022, 6:38 PM
solved the several commands problem, issue is that the requirements from blob storage does not get pulled into the docker container prior to running python -m -prefect.engine, so i can't run the command prior. Is the best practice really to have a seperate image per flow, or one standard image that all your flows use?
z

Zanie

09/23/2022, 6:49 PM
It depends if you’d prefer a slowdown at build time or runtime
In theory, with an image per flow, images will all share common base layers and you can use Docker layer caching to prevent duplicate data transfer.
We’re likely to allow Python dependencies to be installed by the Prefect engine before executing your code sometime in the future.
n

Nic

09/23/2022, 6:51 PM
I will bring this to the attention of the team after the weekend and see what we decide upon for now - thanks for the insight!
r

Ryan Peden

09/24/2022, 1:12 AM
Since you're using
docker-container
, It sounds like the
EXTRA_PIP_PACKAGES
environment variable might help, assuming you are using one of the Prefect base images directly or building atop one? As the docs note, installing the packages at runtime adds overhead and has disadvantages, but you are the best judge of whether it makes sense for your workflow. Setting that environment variable (either during deployment or afterward via the block UI) is a bit less convenient than running pip against requirements.txt, but it might be workable? If you want to try it, you can probably flatten requirements.txt into a
pip install
compatible string you can put in the environment variable by running
awk '{gsub(" ","",$0);printf("%s ",$0)}' requirements.txt
if you're running Linux or MacOS.