https://prefect.io
Join Slack
Is there a way to force the Docker storage to rebuild Python packages from scratch? We're having so...
# ask-community
m
Is there a way to force the Docker storage to rebuild Python packages from scratch? We're having some trouble with cached versions causing problems between flows. Ops is trying to fix it from the CI/CD pipeline side, but I wonder if there's a way to do it from within Prefect?
t
Hi, Matta! Are you talking about docker caching?
m
Yeah! so like the 
python_dependencies
arguments passed to Docker
t
I know that if you supply 
build_kwargs = {"nocache":True}
to the storage object of that will force docker to build without cached layers
m
Ooh, awesome, thanks! I'll try that right now!
Hrm, it still tried to use the cached versions
t
what's the output look like?
m
Copy code
Collecting pyarrow==3.0
  Using cached <https://files.pythonhosted.org/packages/62/d3/a482d8a4039bf931ed6388308f0cc0541d0cab46f0bbff7c897a74f1c576/pyarrow-3.0.0.tar.gz>
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-sdfsfsf/pyarrow/setup.py", line 37, in <module>
        from Cython.Distutils import build_ext as _build_ext
    ModuleNotFoundError: No module named 'Cython'
z
You could try setting the env var 
PIP_NO_CACHE_DIR=true
(with the DockerStorage 
env_vars
dict)
m
Sweet, I'll try that, thanks!
Blah, still doesn't work. I guess the problem is upstream of what I can affect.
(i have no idea how deployment pipelines work 😅 )
13 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by