Thread
#prefect-community
    Dekel R

    Dekel R

    8 months ago
    Hey all, I have a couple of Prefect flows - both local and Vertex runs. All of them are using docker storage. Now the flows work without any problem - but sometimes I do get the following error:
    . 23 January 2022 11:00pm]: 500 Server Error for <http+docker://localhost/v1.41/images/create?tag=2022-01-19t11-58-16-139521-00-00&fromImage=uxxx-flow>: Internal Server Error ("Get "<https://xxx>": context deadline exceeded")
    Seems like the docker registry is unavailable for some reason for a short period of time - exactly when Prefect is trying to pull a flow image and run it. I have 2 questions that I couldn’t find any answers to:1. Is there any configuration option I can use in order to make Prefect retry the image pulling on failure? 2. Can I configure Prefect to pull images only upon change (new flow version) and saving them locally? The way it works right now is that each run invokes an image pull. Thanks!
    Anna Geller

    Anna Geller

    8 months ago
    #1 I haven’t tried that but I believe you can add a container-level restart behavior as follows (docs):
    DockerRun(...,host_config={"restart_policy" = {"Name": "on-failure", "MaximumRetryCount": 3}})
    #2 This is easier to configure on Kubernetes agent (via
    imagePullPolicy
    ) rather than on Docker agent. Afaik, on docker agent you can only generally disable pulling flow images on the agent label by setting
    no_pull=True
    Docs