Dekel R

01/24/2022, 12:59 PM
Hey all, I have a couple of Prefect flows - both local and Vertex runs. All of them are using docker storage. Now the flows work without any problem - but sometimes I do get the following error:
Copy code
. 23 January 2022 11:00pm]: 500 Server Error for <http+docker://localhost/v1.41/images/create?tag=2022-01-19t11-58-16-139521-00-00&fromImage=uxxx-flow>: Internal Server Error ("Get "<https://xxx>": context deadline exceeded")
Seems like the docker registry is unavailable for some reason for a short period of time - exactly when Prefect is trying to pull a flow image and run it. I have 2 questions that I couldn’t find any answers to: 1. Is there any configuration option I can use in order to make Prefect retry the image pulling on failure? 2. Can I configure Prefect to pull images only upon change (new flow version) and saving them locally? The way it works right now is that each run invokes an image pull. Thanks!

Anna Geller

01/24/2022, 1:26 PM
#1 I haven’t tried that but I believe you can add a container-level restart behavior as follows (docs):
Copy code
DockerRun(...,host_config={"restart_policy" = {"Name": "on-failure", "MaximumRetryCount": 3}})
#2 This is easier to configure on Kubernetes agent (via
) rather than on Docker agent. Afaik, on docker agent you can only generally disable pulling flow images on the agent label by setting
👀 1