Hi all, I am trying to run a prefect flow on a dask executor. I am getting the following error `[20...

Matej

05/11/2021, 5:17 PM

Hi all, I am trying to run a prefect flow on a dask executor. I am getting the following error

[2021-05-11 19:11:14+0200] ERROR - prefect.FlowRunner | Unexpected error: ModuleNotFoundError("No module named 'prefect'")

it seems that the worker does not have the prefect installed. Dask is running in a kubernetes cluster so is there a smart way on how to install prefect module on all the dask workers? M

Kevin Kho

05/11/2021, 5:21 PM

Hey @Matej, can you show me how you set up the DaskExecutor/

Kevin Kho

05/11/2021, 5:24 PM

This is how the repo I sent did it: https://github.com/pangeo-forge/pangeo-forge-azure-bakery/blob/add-k8s-cluster/flow_test/manual_flow.py#L34-L46

Matej

05/11/2021, 5:28 PM

I use the following from the tutorials:

executor = DaskExecutor(address="<tcp://dask-scheduler-hostname:8786>")

flow.run(executor=executor)

dask-scheduler-hostname is where the dask scheduler runs in a kubernetes cluster

Kevin Kho

05/11/2021, 5:30 PM

I think if you do it like this, that cluster already exists and it needs to be installed on that cluster whereas the link I sent spins up the cluster and installs the image on it.

Matej

05/11/2021, 5:33 PM

I've used helm install to set up the dask cluster. In the next step i ll look into changing the cluster dynamically. Any simple way on how to pip install prefect on every dask worker ?

Kevin Kho

05/11/2021, 5:44 PM

I think this is what you need

Kevin Kho

05/11/2021, 5:48 PM

You can set

EXTRA_PIP_PACKAGES

and the Dask cluster will install them: https://github.com/dask/dask-docker#environment-variables

Kevin Kho

05/11/2021, 5:49 PM

You can also use the base Prefect image. It has Dask.

Kevin Kho

05/11/2021, 5:49 PM

https://docs.dask.org/en/latest/setup/kubernetes-helm.html#configure-environment

Matej

05/12/2021, 5:01 PM

Hi Kevin, I've set the EXTRA_CONDA_PACKAGES on the worker in the yaml file

Copy code

- name: EXTRA_CONDA_PACKAGES
  value: prefect -c conda-forge

and I still get the same error ModuleNotFound prefect

[2021-05-12 18:57:11+0200] INFO - prefect.DaskExecutor | Stopping executor, waiting for 1 active tasks to complete

[2021-05-12 18:57:11+0200] ERROR - prefect.FlowRunner | Unexpected error: ModuleNotFoundError("No module named 'prefect'")

Traceback (most recent call last):

File "/home/m/repo/prefect/src/prefect/engine/runner.py", line 48, in inner

new_state = method(self, state, *args, **kwargs)

File "/home/m/repo/prefect/src/prefect/engine/flow_runner.py", line 643, in get_flow_run_state

final_states = executor.wait(

File "/home/m/repo/prefect/src/prefect/executors/dask.py", line 414, in wait

return self.client.gather(futures)

File "/home/m/miniconda3/envs/prefect/lib/python3.8/site-packages/distributed/client.py", line 1975, in gather

return self.sync(

File "/home/m/miniconda3/envs/prefect/lib/python3.8/site-packages/distributed/client.py", line 843, in sync

return sync(

File "/home/m/miniconda3/envs/prefect/lib/python3.8/site-packages/distributed/utils.py", line 353, in sync

raise exc.with_traceback(tb)

File "/home/m/miniconda3/envs/prefect/lib/python3.8/site-packages/distributed/utils.py", line 336, in f

result[0] = yield future

File "/home/m/miniconda3/envs/prefect/lib/python3.8/site-packages/tornado/gen.py", line 762, in run

value = future.result()

File "/home/m/miniconda3/envs/prefect/lib/python3.8/site-packages/distributed/client.py", line 1840, in _gather

raise exception.with_traceback(traceback)

File "/opt/conda/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 75, in loads

ModuleNotFoundError: No module named 'prefect'

[2021-05-12 18:57:11+0200] ERROR - <http://prefect.My|prefect.My> First Flow | Unexpected error occured in FlowRunner: ModuleNotFoundError("No module named 'prefect'")

What am i doing wrong? It is a simple hello world example.

Kevin Kho

05/12/2021, 5:04 PM

That looks right. Can I see your config file?

Matej

05/12/2021, 5:05 PM

you mean values.yaml?

Kevin Kho

05/12/2021, 5:07 PM

I thought the link said spin it up first with the values.yaml, and then create the new config.yaml and then use the

upgrade

helm upgrade bald-eel dask/dask -f config.yaml

Kevin Kho

05/12/2021, 5:08 PM

You tried to add it directly to the values.yaml?

Matej

05/12/2021, 5:08 PM

worker.yaml

Matej

05/12/2021, 5:08 PM

yes i upgraded with new values.yaml

Matej

05/12/2021, 5:08 PM

helm upgrade bald-eel dask/dask -f config.yaml

Matej

05/12/2021, 5:08 PM

this is the worker part of the config.yaml

Matej

05/12/2021, 5:10 PM

when i ssh into the worker pod, the module is not installed

Matej

05/12/2021, 5:14 PM

I think the problem is that I have to run dask rootless hence the setting of securityContext for the worker:

Copy code

securityContext:
  runAsUser: 1000
  runAsGroup: 1000

Hence the user 1000 is unable to run conda install because of not root privilege

NotWritableError: The current user does not have write permissions to a required path.

path: /opt/conda/pkgs/urls.txt

uid: 1000

gid: 1000

If you feel that permissions on this path are set incorrectly, you can manually

change them by executing

$ sudo chown 1000:1000 /opt/conda/pkgs/urls.txt

Kevin Kho

05/12/2021, 5:15 PM

Ok i think this might be easier if you use the prefect base image for the workers because that contains dask

Matej

05/12/2021, 5:19 PM

Copy code

prefecthq/prefect:latest

for the worker image?

Kevin Kho

05/12/2021, 5:19 PM

What Python version are you on?

Kevin Kho

05/12/2021, 5:20 PM

For 3.7 yes that looks right

Matej

05/12/2021, 5:21 PM

3.8 is my home python

Matej

05/12/2021, 5:21 PM

*local

Matej

05/12/2021, 5:22 PM

basically how to change the worker.yaml for it to be "prefect base image" ?

Kevin Kho

05/12/2021, 5:27 PM

I think it’s here: https://github.com/dask/helm-chart/blob/main/dask/values.yaml#L64-L71 . Try prefecthq/prefect:latest-python3.8

Kevin Kho

05/12/2021, 5:28 PM

i guess use the same image for the scheduler as well

Matej

05/12/2021, 5:46 PM

Any other way to install prefect on dask workers if the worker is running as NonRoot ?

Kevin Kho

05/12/2021, 5:47 PM

What error did you get in this case?

Matej

05/12/2021, 5:47 PM

the pods crash completely both the scheduler and the workers

Kevin Kho

05/12/2021, 5:48 PM

Any logs or errors?

Matej

05/12/2021, 5:49 PM

Traceback (most recent call last):

File "/usr/local/bin/dask-scheduler", line 5, in <module>

from distributed.cli.dask_scheduler import go

File "/usr/local/lib/python3.8/site-packages/distributed/cli/dask_scheduler.py", line 120, in <module>

def main(

File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 247, in decorator

_param_memo(f, OptionClass(param_decls, **option_attrs))

File "/usr/local/lib/python3.8/site-packages/click/core.py", line 2467, in __init__

super().__init__(param_decls, type=type, multiple=multiple, **attrs)

File "/usr/local/lib/python3.8/site-packages/click/core.py", line 2106, in __init__

raise ValueError(

ValueError: 'default' must be a list when 'multiple' is true.

shceduler pod logs

Matej

05/12/2021, 5:49 PM

(base) m@desktop:~/repo/kubernetes/dask$ k logs dask-worker-7b55c8bb7-4fsft

Traceback (most recent call last):

File "/usr/local/bin/dask-worker", line 8, in <module>

sys.exit(go())

File "/usr/local/lib/python3.8/site-packages/distributed/cli/dask_worker.py", line 461, in go

check_python_3()

File "/usr/local/lib/python3.8/site-packages/distributed/cli/utils.py", line 32, in check_python_3

_unicodefun._verify_python3_env()

AttributeError: module 'click._unicodefun' has no attribute '_verify_python3_env'

worker pods

Matej

05/12/2021, 5:51 PM

thanks for your patience

Kevin Kho

05/12/2021, 5:54 PM

Sure! Can you try this image instead:

prefecthq/prefect:0.14.17-python3.8

. Just dropping the version down

Matej

05/12/2021, 5:57 PM

in the tag: correct?

Kevin Kho

05/12/2021, 5:57 PM

yes that’s right

Kevin Kho

05/12/2021, 5:57 PM

0.14.17-python3.8 is tha tag

Matej

05/12/2021, 5:59 PM

it is running. I ll test the hello world.

Matej

05/12/2021, 6:02 PM

Perfect! now it works! thanks so much. What do you suggest I can do to add additional packages such as numpy or pandas to the workers in the case that I have NonRoot pod? Thanks.

Kevin Kho

05/12/2021, 6:03 PM

Nice! If this approach doesn’t work, I guess the approach is to make a new image and host it somewhere like Dockerhub to pull down.

Matej

05/12/2021, 6:03 PM

aha so I prepare my own image with all the required packages and then just set it up in the same way.

Matej

05/12/2021, 6:04 PM

because now even my dashboard does not work

Dask needs bokeh >= 0.13.0 for the dashboard.

Kevin Kho

05/12/2021, 6:04 PM

Yes it seems like that would work

Matej

05/12/2021, 6:05 PM

perhaps maybe i can set up some worker preload script that would set up a non root conda where a non root user can install?

Kevin Kho

05/12/2021, 6:05 PM

Maybe that is worth a shot

Matej

05/12/2021, 6:06 PM

ok ill try. thanks for help

Matej

05/13/2021, 5:27 AM

Any idea why the newest prefect base image fails? (I wanted to try artifact store) 0.14.19-python3.7 fails on scheduler and worker with the following exception

nvidia 34041856 87 nvidia_modeset,nvidia_uvm, Live 0xffffffffc0ce0000 (POE)

Traceback (most recent call last):

File "/usr/local/bin/dask-scheduler", line 5, in <module>

from distributed.cli.dask_scheduler import go

File "/usr/local/lib/python3.7/site-packages/distributed/cli/dask_scheduler.py", line 119, in <module>

@click.version_option()

File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 247, in decorator

_param_memo(f, OptionClass(param_decls, **option_attrs))

File "/usr/local/lib/python3.7/site-packages/click/core.py", line 2467, in __init__

super().__init__(param_decls, type=type, multiple=multiple, **attrs)

File "/usr/local/lib/python3.7/site-packages/click/core.py", line 2108, in __init__

) from None

ValueError: 'default' must be a list when 'multiple' is true.

Kevin Kho

05/13/2021, 1:33 PM

Yeah this issue was just fixed. It's because of version conflicts with

click

. The CLI package

Matej

05/13/2021, 4:30 PM

so 0.14.19-python3.7 works now right?

Kevin Kho

05/13/2021, 4:33 PM

No I don’t think so because the PR to set an upper bound on the click version didn’t get released yet

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.