Hi there, We are using Prefect Cloud to handle our workflow management. In the process of standing it up, our deployment recipe uses a Kubernetes Agent and Docker Storage. When registering a flow to cloud, without specifying an environment, the flow is submitted for execution, and runs entirely fine. However, for long running map tasks, we’d like to consider using

DaskKubernetesEnvironment

. Perhaps, we can use

DaskKubernetesEnvironment

for all of our flows. Upon registering a new flow, as shown below, to the cloud and submitting a Quick Run, we get the error: 

Kubernetes Error: Back-off pulling image

When not specifying the

DaskKubernetesEnvironment

all registering, deploying, flow execution works just fine. Here is a sample flow that I am trying to use Dask for:

with Flow("test-flow") as flow:
  numbers = numbers_task()
  first_map = map_task.map(numbers)
  second_map = map_task.map(first_map)
  reduction = reduce_task(second_map)

flow.storage = Docker(
  registry_url="<http://parkmobile-docker.jfrog.io|parkmobile-docker.jfrog.io>",
  image_name="test-flow",
  image_tag="0.0.1"
)

flow.environment = DaskKubernetesEnvironment(min_workers=2, max_workers=4)

flow.register("test")

Any ideas as to why the DaskKubernetesEnvironment is throwing off the flow execution?

Hi Mitchell - would you mind moving this code snippet into the thread? It’s large so it takes up the whole chat window. Otherwise, how is your agent authenticating with your Docker registry? If it’s through a K8s secret, you probably need to mount the same secret to your daskK8s environment (There should be a keyword argument for that)

So you are saying that there needs to be a separate daskk8 environment with a separate agent?

No, nothing that extreme; the DaskKubernetesEnvironment requires the ability to create new K8s jobs with your Flow’s Docker image, which requires that it have the ability to pull from your docker registry. Typically, docker registries are authenticated via K8s secrets (let me know if you use something different). Check out the docs here that discuss the

image_pull_secret

kwarg: https://docs.prefect.io/api/latest/environments/execution.html#daskkubernetesenvironment

@Kenan Alptekin let me know if you have some additional context, but i believe we authenticated via

prefect agent start kubernetes --env IMAGE_PULL_SECRETS=my-img-pull-secret

Perfect, so then that doc explains how to pass that same secret onto your Dask K8s Environment by name via the

image_pull_secret

kwarg

I see, so this would need to be passed via

kwarg

yup, that’s accurate. Environments are getting a large overhaul in the coming weeks, so these sorts of issues will become much more straightforward to debug