I am running a prefect server executing with a Dask Kubernetes cluster, and I'm runinng the hello world test script and keep getting this error.
python test.py
[2021-05-19 21:06:50+0000] INFO - prefect.FlowRunner | Beginning Flow run for 'Run Me'
[2021-05-19 21:06:50+0000] INFO - prefect.DaskExecutor | Connecting to an existing Dask cluster at
tcp://x.x.x.x:8786
/home/dani1057/anaconda3/envs/faraday3.8/lib/python3.8/site-packages/distributed/client.py:1140: VersionMismatchWarning: Mismatched versions found
+---------+---------------+---------------+---------------+
| Package | client | scheduler | workers |
+---------+---------------+---------------+---------------+
| python | 3.8.8.final.0 | 3.8.0.final.0 | 3.8.0.final.0 |
+---------+---------------+---------------+---------------+
warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))
[2021-05-19 21:06:50+0000] ERROR - prefect.FlowRunner | Unexpected error: ModuleNotFoundError("No module named 'prefect'")
Traceback (most recent call last):
File "/home/dani1057/anaconda3/envs/faraday3.8/lib/python3.8/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/home/dani1057/anaconda3/envs/faraday3.8/lib/python3.8/site-packages/prefect/engine/flow_runner.py", line 643, in get_flow_run_state
final_states = executor.wait(
File "/home/dani1057/anaconda3/envs/faraday3.8/lib/python3.8/site-packages/prefect/executors/dask.py", line 414, in wait
return self.client.gather(futures)
File "/home/dani1057/anaconda3/envs/faraday3.8/lib/python3.8/site-packages/distributed/client.py", line 1975, in gather
return self.sync(
File "/home/dani1057/anaconda3/envs/faraday3.8/lib/python3.8/site-packages/distributed/client.py", line 843, in sync
return sync(
File "/home/dani1057/anaconda3/envs/faraday3.8/lib/python3.8/site-packages/distributed/utils.py", line 353, in sync
raise exc.with_traceback(tb)
File "/home/dani1057/anaconda3/envs/faraday3.8/lib/python3.8/site-packages/distributed/utils.py", line 336, in f
result[0] = yield future
File "/home/dani1057/anaconda3/envs/faraday3.8/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/home/dani1057/anaconda3/envs/faraday3.8/lib/python3.8/site-packages/distributed/client.py", line 1840, in _gather
raise exception.with_traceback(traceback)
File "/opt/conda/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 75, in loads
ModuleNotFoundError: No module named 'prefect'
[2021-05-19 21:06:50+0000] ERROR - prefect.Run Me | Unexpected error occured in FlowRunner: ModuleNotFoundError("No module named 'prefect'")
[2:15 PM] I also have a kubernetes agent running on the server
[2:16 PM] Does the agent have to be running on the cluster?