https://prefect.io logo
m

Mitchell Bregman

10/06/2020, 4:27 PM
Hi there, We are using Prefect Cloud to handle our workflow management. In the process of standing it up, our deployment recipe uses a Kubernetes Agent and Docker Storage. When registering a flow to cloud, without specifying an environment, the flow is submitted for execution, and runs entirely fine. However, for long running map tasks, we’d like to consider using
DaskKubernetesEnvironment
. Perhaps, we can use
DaskKubernetesEnvironment
for all of our flows. Upon registering a new flow, as shown below, to the cloud and submitting a Quick Run, we get the error: 
Copy code
Kubernetes Error: Back-off pulling image
When not specifying the
DaskKubernetesEnvironment
all registering, deploying, flow execution works just fine. Here is a sample flow that I am trying to use Dask for:
Copy code
with Flow("test-flow") as flow:
  numbers = numbers_task()
  first_map = map_task.map(numbers)
  second_map = map_task.map(first_map)
  reduction = reduce_task(second_map)

flow.storage = Docker(
  registry_url="<http://parkmobile-docker.jfrog.io|parkmobile-docker.jfrog.io>",
  image_name="test-flow",
  image_tag="0.0.1"
)

flow.environment = DaskKubernetesEnvironment(min_workers=2, max_workers=4)

flow.register("test")
Any ideas as to why the DaskKubernetesEnvironment is throwing off the flow execution?
c

Chris White

10/06/2020, 4:37 PM
Hi Mitchell - would you mind moving this code snippet into the thread? It’s large so it takes up the whole chat window. Otherwise, how is your agent authenticating with your Docker registry? If it’s through a K8s secret, you probably need to mount the same secret to your daskK8s environment (There should be a keyword argument for that)
m

Mitchell Bregman

10/06/2020, 4:49 PM
So you are saying that there needs to be a separate daskk8 environment with a separate agent?
I figured I could use the single Kubernetes Agent i have stood up, would spawn off some sort of Dask environment for me, and then execute there
Let me know if I’m thinking about it incorrectly
c

Chris White

10/06/2020, 4:51 PM
No, nothing that extreme; the DaskKubernetesEnvironment requires the ability to create new K8s jobs with your Flow’s Docker image, which requires that it have the ability to pull from your docker registry. Typically, docker registries are authenticated via K8s secrets (let me know if you use something different). Check out the docs here that discuss the
image_pull_secret
kwarg: https://docs.prefect.io/api/latest/environments/execution.html#daskkubernetesenvironment
m

Mitchell Bregman

10/06/2020, 4:54 PM
@Kenan Alptekin let me know if you have some additional context, but i believe we authenticated via
prefect agent start kubernetes --env IMAGE_PULL_SECRETS=my-img-pull-secret
c

Chris White

10/06/2020, 4:55 PM
Perfect, so then that doc explains how to pass that same secret onto your Dask K8s Environment by name via the
image_pull_secret
kwarg
m

Mitchell Bregman

10/06/2020, 4:56 PM
I see, so this would need to be passed via
kwarg
Ideally, i was hoping that
--env IMAGE_PULL_SECRETS=my-img-pull-secret
would handle the rest
but my knowledge of how k8s work is quite minimal.. will touch back any findings!
c

Chris White

10/06/2020, 4:57 PM
yup, that’s accurate. Environments are getting a large overhaul in the coming weeks, so these sorts of issues will become much more straightforward to debug
🙌 1