Is there a way to prevent the prefect docker flow storage from pip installing the latest and greates...

Dolor Oculus

01/14/2021, 2:21 PM

Is there a way to prevent the prefect docker flow storage from pip installing the latest and greatest prefect? My build setup/team is aghast. (We prefer to explicitly manage dependencies ourselves.)

Spencer

01/14/2021, 2:38 PM

According to the source, you'll need to pre-install prefect. https://github.com/PrefectHQ/prefect/blob/master/src/prefect/storage/docker.py#L204-L210

Billy McMonagle

01/14/2021, 3:07 PM

In my build environment, I have

pip install prefect==PREFECT_VERSION

, and set explicitly the base image that docker storage is built from. Together these seem to work well.

Chris White

01/14/2021, 3:58 PM

The Docker storage interface determines what version to install based on your local version, so if it’s installing

master

or the latest version, that must be what you’re running on locally. If you want to specify a different version, there is a

prefect_version

kwarg to the interface that you can use to pin to whatever version you’d like

Dolor Oculus

01/15/2021, 4:09 PM

@Chris White 🔥 thank you

Dolor Oculus

01/15/2021, 4:10 PM

we're using conda though, so pip is still an issue.

Dolor Oculus

01/15/2021, 4:11 PM

I'll share the above with my build team though and see if they have thoughts.

Kevin Weiler

01/15/2021, 4:23 PM

yeah - to be clear @Chris White - we specifically DON’T want prefect installing anything in our environment. There are 2 reasons for this: 1. the container we base off has a conda environment (that we build) that already has prefect installed 2. because the environment is conda, it’s possible for pip to install dependencies over top of ours.

Kevin Weiler

01/15/2021, 4:24 PM

a bit of further explanation: We don’t particularly like conda, but until very recently, it was the only game in town as far as actual package resolution is concerned. Pip (prior to v20.3) doesn’t actually handle transitive dependencies, which is a huge problem in a sufficiently complex environment. As an example, if package A depends on package Z==2 and Package B depends on package Z==1, pip installs the transitive dependencies in whatever order it installs the explicit dependencies. So if package A is listed first in your

requirements.txt

then it installs package Z==2 first and then Z==1 over top of it second.

😱 1

Kevin Weiler

01/15/2021, 4:25 PM

here’s a bit more reading on the topic https://pyfound.blogspot.com/2020/11/pip-20-3-new-resolver.html

8 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.