Mitchell Bregman
10/06/2020, 4:27 PMDaskKubernetesEnvironment
. Perhaps, we can use DaskKubernetesEnvironment
for all of our flows. Upon registering a new flow, as shown below, to the cloud and submitting a Quick Run, we get the error:
Kubernetes Error: Back-off pulling image
When not specifying the DaskKubernetesEnvironment
all registering, deploying, flow execution works just fine. Here is a sample flow that I am trying to use Dask for:
with Flow("test-flow") as flow:
numbers = numbers_task()
first_map = map_task.map(numbers)
second_map = map_task.map(first_map)
reduction = reduce_task(second_map)
flow.storage = Docker(
registry_url="<http://parkmobile-docker.jfrog.io|parkmobile-docker.jfrog.io>",
image_name="test-flow",
image_tag="0.0.1"
)
flow.environment = DaskKubernetesEnvironment(min_workers=2, max_workers=4)
flow.register("test")
Any ideas as to why the DaskKubernetesEnvironment is throwing off the flow execution?Chris White
Mitchell Bregman
10/06/2020, 4:49 PMChris White
image_pull_secret
kwarg: https://docs.prefect.io/api/latest/environments/execution.html#daskkubernetesenvironmentMitchell Bregman
10/06/2020, 4:54 PMprefect agent start kubernetes --env IMAGE_PULL_SECRETS=my-img-pull-secret
Chris White
image_pull_secret
kwargMitchell Bregman
10/06/2020, 4:56 PMkwarg
--env IMAGE_PULL_SECRETS=my-img-pull-secret
would handle the restChris White