Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
t
Thomas Agung Santoso
05/31/2022, 2:31 AMHi everyone. This morning I encountered problem when running prefect workflow. the log say : RuntimeError: Missing dependency kubectl. Please install kubectl following the instructions for your OS. This never happened before, I try to update kubectl at the worker cluster and not helping. I'm using dask on kubernetes for executor. any idea?
k
Kevin Kho
05/31/2022, 2:39 AMI think you are using Kuberentes agent not in a cluster? The Kubernetes agent needs kubectl
t
Thomas Agung Santoso
05/31/2022, 2:46 AMPrefect agent is installed in the same cluster.
k
Kevin Kho
05/31/2022, 2:50 AMFor some reason, the error says you don’t have kubectl there. What base image did you use?
t
Thomas Agung Santoso
05/31/2022, 2:59 AMI used "prefecthq/prefect:latest-python3.8". I attach the captured logs from prefect agent too.
[2022-05-31 03:17:11+0000] INFO - prefect.CloudFlowRunner | Beginning Flow run for 'endspider_category_pipeline'
[2022-05-31 03:17:13+0000] INFO - prefect.DaskExecutor | Creating a new Dask cluster with...<flow>.<lambda>
Creating scheduler pod on cluster. This may take some time.
[2022-05-31 03:17:16+0000] ERROR - prefect.CloudFlowRunner | Unexpected error: RuntimeError('Missing dependency kubectl. Please install kubectl following the instructions for your OS. ')
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prefect/engine/flow_runner.py", line 442, in get_flow_run_state
with self.check_for_cancellation(), executor.start():
File "/usr/local/lib/python3.8/contextlib.py", line 113, in enter
return next(self.gen)
File "/usr/local/lib/python3.8/site-packages/prefect/executors/dask.py", line 243, in start
with self.cluster_class(**self.cluster_kwargs) as cluster:
File "/home/administrator/PROJECT/endlab/endspider/category_pipeline.py", line 147, in <lambda>
File "/usr/local/lib/python3.8/site-packages/dask_kubernetes/classic/kubecluster.py", line 498, in init
super().__init__(**self.kwargs)
File "/usr/local/lib/python3.8/site-packages/distributed/deploy/spec.py", line 260, in init
self.sync(self._start)
File "/usr/local/lib/python3.8/site-packages/distributed/utils.py", line 320, in sync
return sync(
File "/usr/local/lib/python3.8/site-packages/distributed/utils.py", line 387, in sync
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.8/site-packages/distributed/utils.py", line 360, in f
result = yield future
File "/usr/local/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/usr/local/lib/python3.8/site-packages/dask_kubernetes/classic/kubecluster.py", line 636, in _start
self.forwarded_dashboard_port = await port_forward_dashboard(
File "/usr/local/lib/python3.8/site-packages/dask_kubernetes/common/networking.py", line 107, in port_forward_dashboard
port = await port_forward_service(service_name, namespace, 8787)
File "/usr/local/lib/python3.8/site-packages/dask_kubernetes/common/networking.py", line 72, in port_forward_service
check_dependency("kubectl")
File "/usr/local/lib/python3.8/site-packages/dask_kubernetes/common/utils.py", line 38, in check_dependency
raise RuntimeError(
RuntimeError: Missing dependency kubectl. Please install kubectl following the instructions for your OS.
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f7bbe341af0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f7bbe341eb0>
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7f7bbe355220>, 8337662.611438069)]']
connector: <aiohttp.connector.TCPConnector object at 0x7f7bbe341850>
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7f7bbe355dc0>, 8337662.606091255)]']
connector: <aiohttp.connector.TCPConnector object at 0x7f7bbe341c40>
that is logs from prefect job pods that failed.
l
Lucas Hosoya
05/31/2022, 2:22 PMHi, are there any Taints/Tolerations in your k8s nodes? This might be the issue, since it happened to me just after we added taints