https://prefect.io logo
Title
t

Thomas Agung Santoso

05/31/2022, 2:31 AM
Hi everyone. This morning I encountered problem when running prefect workflow. the log say : RuntimeError: Missing dependency kubectl. Please install kubectl following the instructions for your OS. This never happened before, I try to update kubectl at the worker cluster and not helping. I'm using dask on kubernetes for executor. any idea?
k

Kevin Kho

05/31/2022, 2:39 AM
I think you are using Kuberentes agent not in a cluster? The Kubernetes agent needs kubectl
t

Thomas Agung Santoso

05/31/2022, 2:46 AM
Prefect agent is installed in the same cluster.
k

Kevin Kho

05/31/2022, 2:50 AM
For some reason, the error says you don’t have kubectl there. What base image did you use?
t

Thomas Agung Santoso

05/31/2022, 2:59 AM
I used "prefecthq/prefect:latest-python3.8". I attach the captured logs from prefect agent too.
[2022-05-31 03:17:11+0000] INFO - prefect.CloudFlowRunner | Beginning Flow run for 'endspider_category_pipeline'
[2022-05-31 03:17:13+0000] INFO - prefect.DaskExecutor | Creating a new Dask cluster with
<flow>.<lambda>
...
Creating scheduler pod on cluster. This may take some time.
[2022-05-31 03:17:16+0000] ERROR - prefect.CloudFlowRunner | Unexpected error: RuntimeError('Missing dependency kubectl. Please install kubectl following the instructions for your OS. ')
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prefect/engine/flow_runner.py", line 442, in get_flow_run_state
with self.check_for_cancellation(), executor.start():
File "/usr/local/lib/python3.8/contextlib.py", line 113, in enter
return next(self.gen)
File "/usr/local/lib/python3.8/site-packages/prefect/executors/dask.py", line 243, in start
with self.cluster_class(**self.cluster_kwargs) as cluster:
File "/home/administrator/PROJECT/endlab/endspider/category_pipeline.py", line 147, in <lambda>
File "/usr/local/lib/python3.8/site-packages/dask_kubernetes/classic/kubecluster.py", line 498, in init
super().__init__(**self.kwargs)
File "/usr/local/lib/python3.8/site-packages/distributed/deploy/spec.py", line 260, in init
self.sync(self._start)
File "/usr/local/lib/python3.8/site-packages/distributed/utils.py", line 320, in sync
return sync(
File "/usr/local/lib/python3.8/site-packages/distributed/utils.py", line 387, in sync
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.8/site-packages/distributed/utils.py", line 360, in f
result = yield future
File "/usr/local/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/usr/local/lib/python3.8/site-packages/dask_kubernetes/classic/kubecluster.py", line 636, in _start
self.forwarded_dashboard_port = await port_forward_dashboard(
File "/usr/local/lib/python3.8/site-packages/dask_kubernetes/common/networking.py", line 107, in port_forward_dashboard
port = await port_forward_service(service_name, namespace, 8787)
File "/usr/local/lib/python3.8/site-packages/dask_kubernetes/common/networking.py", line 72, in port_forward_service
check_dependency("kubectl")
File "/usr/local/lib/python3.8/site-packages/dask_kubernetes/common/utils.py", line 38, in check_dependency
raise RuntimeError(
RuntimeError: Missing dependency kubectl. Please install kubectl following the instructions for your OS.
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f7bbe341af0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f7bbe341eb0>
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7f7bbe355220>, 8337662.611438069)]']
connector: <aiohttp.connector.TCPConnector object at 0x7f7bbe341850>
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7f7bbe355dc0>, 8337662.606091255)]']
connector: <aiohttp.connector.TCPConnector object at 0x7f7bbe341c40>
that is logs from prefect job pods that failed.
l

Lucas Hosoya

05/31/2022, 2:22 PM
Hi, are there any Taints/Tolerations in your k8s nodes? This might be the issue, since it happened to me just after we added taints