https://prefect.io
Join Slack
I have an issue with building a docker image which I am pushing to Prefect Cloud. I have managed to ...
# ask-community
e
I have an issue with building a docker image which I am pushing to Prefect Cloud. I have managed to successfully run the flow locally on my machine, the issue I am having is creating the Docker image to push to the Prefect cloud. It basically fails to build the image when I use the 
prefect register
 method, during building of the custom docker image on the local machine. I can build the image using the underlying docker command, but only if I use the flag 
--network=host
in the build call. I have been successful in doing this before, I am using a newly installed VM for the submission, and so I am left wondering if there are default settings required for Docker. Is there a flag or setting that needs to be added somewhere, either to Docker or to Prefect, that enables the container to access the external network and internet during build time?
a
can you add that --network=host when you are starting agent eg prefect agent docker start --network=host
e
@Antti Tupamäki Thanks for the suggestions, but as far as my understanding goes, I am not wanting to execute the flow locally, I am simply building the docker container for pushing to the prefect cloud. From there, it will be orchestrated onto a Kubernetes cluster for execution. It is the building of the image locally prior to pushing to the Prefect Cloud where I am encountering network issues.
k
Hey @Ed Morris, this is the first time I’ve seen this around and am not sure what causes this. I think the setting is Docker-specific rather than Prefect as prefect basically just uses whatever Docker installation you have. If you still have problems building though, could you check if 
docker build … --network=host
will help you with that, and then we can figure out how to add it on the Prefect side? But I haven’t seen this kind of error. I think it would also help if you could exec into the container without 
--network=host
and try a simple API request with the 
urllib3
. From experience, the newer versions of this library have these kinds of issues and downgrading may help. If you have Prefect working already somewhere else, I think it would help to compare the versions of this library
Also, we’d appreciate it if you could maybe move come of those logs over to the thread to keep the main channel more compact when you get the chance. Some community users like to browse the threads. 😅
👍 1
e
The problem in detail: Basically, I instantiate a building of the Prefect flow docker image by using the Prefect CLI as follows:
Copy code
prefect register --project Examples -m "cub_tools_prefect.flows.cub_tools_train_flow"
This kicks off the compiling of the docker image ready for pushing it to our container registry. The issue I have is that the docker installation fails when trying to install packages on the container as part of the container building process. I get the following errors:
Copy code
Collecting flows...
Azure ML SDK version: 1.34.0
MLOpsConfiguration.load() called
[2021-09-29 11:44:58+0000] INFO - prefect | No ENV_FOR_DYNACONF environment variable found!!!
MLOpsConfiguration._setup_basic_env_vars returned false. No secrets will be loaded from vaults.
Processing 'cub_tools_prefect.flows.cub_tools_train_flow':
  Building `Docker` storage...
[2021-09-29 11:44:58+0000] INFO - prefect.Docker | Building the flow's Docker storage...
Step 1/19 : FROM prefecthq/prefect:0.15.6-python3.8
 ---> 50e39b83641e
Step 2/19 : RUN python -m pip install --upgrade pip
 ---> Running in ef55ed5ed26f
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f932275beb0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pip/
I have replicated this issue by pulling the prefect base docker image, and running the same command inside the interactive terminal:
Copy code
(py38_azureml_prefect_cub200) edmorris@ecm-dev-gpu-001:~/projects/image_classification/Caltech_Birds_Prefect$ docker run -t -i prefecthq/prefect:0.15.6-python3.8
            _____  _____  ______ ______ ______ _____ _______
           |  __ \|  __ \|  ____|  ____|  ____/ ____|__   __|
           | |__) | |__) | |__  | |__  | |__ | |       | |
           |  ___/|  _  /|  __| |  __| |  __|| |       | |
           | |    | | \ \| |____| |    | |___| |____   | |
           |_|    |_|  \_\______|_|    |______\_____|  |_|

Thanks for using Prefect!!!

This is the official docker image for Prefect Core, intended for executing
Prefect Flows. For more information, please see the docs:
<https://docs.prefect.io/core/getting_started/installation.html#docker>

root@e920510dd437:/# python -m pip install --upgrade pip
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f949673dd60>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pip/
This problem is solved by adding the 
--network=host
flag to the docker run command as follows:
Copy code
(py38_azureml_prefect_cub200) edmorris@ecm-dev-gpu-001:~/projects/image_classification/Caltech_Birds_Prefect$ docker run -t -i --network=host prefecthq/prefect:0.15.6-python3.8
            _____  _____  ______ ______ ______ _____ _______
           |  __ \|  __ \|  ____|  ____|  ____/ ____|__   __|
           | |__) | |__) | |__  | |__  | |__ | |       | |
           |  ___/|  _  /|  __| |  __| |  __|| |       | |
           | |    | | \ \| |____| |    | |___| |____   | |
           |_|    |_|  \_\______|_|    |______\_____|  |_|

Thanks for using Prefect!!!

This is the official docker image for Prefect Core, intended for executing
Prefect Flows. For more information, please see the docs:
<https://docs.prefect.io/core/getting_started/installation.html#docker>

root@ecm-dev-gpu-001:/# python -m pip install --upgrade pip
Collecting pip
  Downloading pip-21.2.4-py3-none-any.whl (1.6 MB)
     |████████████████████████████████| 1.6 MB 18.2 MB/s
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.4
    Uninstalling pip-20.2.4:
      Successfully uninstalled pip-20.2.4
Successfully installed pip-21.2.4
k
Thank you!
e
@Kevin Kho, I have successfully managed to build the image by using the docker command: 
docker build --network=host ./ -f Dockerfile-Prefect
If I remove the 
--network=host
flag, then I replicate the issue when the image is built with the 
prefect register
call.
k
Gotcha. Ok I’ll see how we can add that to the build on the Prefect side
e
I don't really understand why it's suddenly become an issue though, as I am more or less sure it's not a default I had changed consciously in previous installs of docker / prefect. I am relatively new to Prefect, and wouldn't class myself as an expert in using docker either, so it's difficult for me to be completely sure on these thinfs.
k
So to replicate that in Prefect Docker storage, you would do:
Copy code
flow.storage = Docker(..., build_kwargs={"network_mode":"host"})
I am not an expert on Docker either, but this is more of a Docker and networking thing than the Prefect side because Prefect just uses 
dockerpy
which just hits your installation. I wouldn’t know though what setting would cause this on the Docker side. How did you know to add 
--network=host
? I can try digging a bit with that as a starting point
e
@Kevin Kho, so in the flow file, where would that come?
k
When defining the storage for the Flow. I assume you are using Docker storage?
e
We build the image and use a Azure Container Repository to host the flow.
k
Copy code
with Flow(...) as flow:
     ...

flow.storage = Docker(...)
How do you define your flow storage?
e
Actually it was buried in boiler plate code we have and I'd never looked into that function. 🙂 You are correct, we use docker storage.
k
Ah I see…well you must if 
register
is triggering a build haha. You should need to add it to that function
e
Right, I get this now.
In the Docker class call, I can use the build_kwargs as you suggested. This has recently been refactored our end, so there's every possibility this has been missed off during the refactoring!
k
Ahh that may be possible
e
Thank you @Kevin Kho, I think this will be the answer, as I know docker will build the image if the correct flag is given.
👍 1
@Kevin Kho just to confirm, inclusion of the network=host build argument using the 
build_kwargs
argument of the 
Docker
class, which was used during the 
prefect register
call, was successful in building custom docker images locally and then pushing them to a container repository, registering the flow with the prefeft cloud and executing on Kubernetes. Thank you.
k
Thanks for circling back and letting me know!
👍 1
95 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by