Tom Klein
07/10/2022, 9:54 AMUnexpected error: RuntimeError('Missing dependency kubectl. Please install kubectl following the instructions for your OS. ')
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/prefect/engine/flow_runner.py", line 442, in get_flow_run_state
with self.check_for_cancellation(), executor.start():
File "/usr/lib/python3.9/contextlib.py", line 119, in __enter__
return next(self.gen)
File "/usr/lib/python3.9/site-packages/prefect/executors/dask.py", line 243, in start
with self.cluster_class(**self.cluster_kwargs) as cluster:
File "<string>", line 215, in <lambda>
File "/usr/lib/python3.9/site-packages/dask_kubernetes/classic/kubecluster.py", line 503, in __init__
super().__init__(**self.kwargs)
File "/usr/lib/python3.9/site-packages/distributed/deploy/spec.py", line 264, in __init__
self.sync(self._start)
File "/usr/lib/python3.9/site-packages/distributed/utils.py", line 336, in sync
return sync(
File "/usr/lib/python3.9/site-packages/distributed/utils.py", line 403, in sync
raise exc.with_traceback(tb)
File "/usr/lib/python3.9/site-packages/distributed/utils.py", line 376, in f
result = yield future
File "/usr/lib/python3.9/site-packages/tornado/gen.py", line 769, in run
value = future.result()
File "/usr/lib/python3.9/site-packages/dask_kubernetes/classic/kubecluster.py", line 647, in _start
self.forwarded_dashboard_port = await port_forward_dashboard(
File "/usr/lib/python3.9/site-packages/dask_kubernetes/common/networking.py", line 110, in port_forward_dashboard
port = await port_forward_service(service_name, namespace, 8787)
File "/usr/lib/python3.9/site-packages/dask_kubernetes/common/networking.py", line 75, in port_forward_service
check_dependency("kubectl")
File "/usr/lib/python3.9/site-packages/dask_kubernetes/common/utils.py", line 38, in check_dependency
raise RuntimeError(
RuntimeError: Missing dependency kubectl. Please install kubectl following the instructions for your OS.
kubernetes
as an extra for the prefect pip installation, not enough?with Flow(
FLOW_NAME,
result=s3_result,
executor=DaskExecutor(
cluster_class=lambda: KubeCluster(make_pod_spec(image=prefect.context.image)),
adapt_kwargs={"minimum": 2, "maximum": 2},
),
run_config=KubernetesRun(
image="xyz_my_image",
memory_request="32Gi",
env=my_env,
),
) as flow:
curl -LO <https://dl.k8s.io/release/v1.21.2/bin/linux/amd64/kubectl> &&\
install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl &&\
lines to our docker image?
i thought that Prefect comes with the necessary Dask/Kubernetes dependencies?Anna Geller
07/10/2022, 10:29 AMTom Klein
07/10/2022, 10:31 AMdask-kubernetes
packaged along at least the Kubernetes
extra (and that the latter would install kubectl if necessary)Anna Geller
07/10/2022, 10:35 AMTom Klein
07/10/2022, 10:38 AMkubectl
was successfully installed but seems like it’s missing config:
Step 12/23 : RUN kubectl version
---> Running in 6f3b72feadad
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
The connection to the server localhost:8080 was refused - did you specify the right host or port?
Client Version: <http://version.Info|version.Info>{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:22:29Z", GoVersion:"go1.18.3", Compiler:"gc", Plat
form:"linux/amd64"}
Kustomize Version: v4.5.4
[SYSTEM]
Message Failed to build image: honeybook/ds-lead-enrichment:2022
Caused by Container for step title: Building Docker Image, step type: build, operation: Building image.
this is what i don’t get - do we need to handle these low-level configs as part of working with prefect? i thought prefect takes care of the interaction with the kubernetes cluster for us (e.g. we didn’t have this issue when running Kubernetes tasks)Anna Geller
07/10/2022, 11:55 AMTom Klein
07/10/2022, 12:00 PMaws cli
because it supposedly gives the ability to fetch/create a config file for kubectl
automatically - and now it’s failing because despite our Prefect deployments (e.g. the agent, the jobs) having a k8s servicerole with full AWS permissions, it still tells me:
Removing intermediate container b69c69743ae3
---> 4c2d52aa2e3a
Step 13/24 : RUN aws eks update-kubeconfig --region us-east-1 --name prefect
---> Running in deb16c5fa885
Unable to locate credentials. You can configure credentials by running "aws configure".
i just hoped that we wouldn’t have to get to these kind of low-level configurations when using Prefect and that once we set it up it simply integrates with what we need (given that it has the permissions)
i thought it’s mostly a matter of giving it an executor with an integer representing the number of workers and an imageRunNamespacedJob
) i didn’t have to do any of that stuffAnna Geller
07/10/2022, 12:02 PMTom Klein
07/10/2022, 12:02 PMAnna Geller
07/10/2022, 12:04 PMTom Klein
07/10/2022, 12:05 PMENTRYPOINT
for docker?Anna Geller
07/10/2022, 12:18 PMTom Klein
07/10/2022, 12:19 PMAnna Geller
07/10/2022, 12:19 PM- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1
Tom Klein
07/10/2022, 12:20 PMkubectl
prefect.context.image
is a legitimate reference? cause i don’t see it in the docs:
https://docs.prefect.io/api/latest/utilities/context.html#context-2value = future.result()
File "/usr/lib/python3.9/site-packages/dask_kubernetes/classic/kubecluster.py", line 647, in _start
self.forwarded_dashboard_port = await port_forward_dashboard(
File "/usr/lib/python3.9/site-packages/dask_kubernetes/common/networking.py", line 110, in port_forward_dashboard
port = await port_forward_service(service_name, namespace, 8787)
File "/usr/lib/python3.9/site-packages/dask_kubernetes/common/networking.py", line 94, in port_forward_service
raise ConnectionError("kubectl port forward failed")
ConnectionError: kubectl port forward failed
i understand this is more of a dask-kubernetes issue than a prefect issue — but any chance you have any idea what’s happening?Anna Geller
07/10/2022, 12:54 PMTom Klein
07/10/2022, 12:59 PMno-dahsboard
would help? the pod is created with everything — i suspect the issue might be more fundamentalOliver Mannion
07/10/2022, 1:33 PMTom Klein
07/10/2022, 1:34 PMOliver Mannion
07/10/2022, 1:41 PMkubectl get service
?Tom Klein
07/10/2022, 1:41 PMkubectl
myself, only our devops have permissions for that, this ^^ is what they sent me to show that the pod was created)ClusterIP
?Anna Geller
07/10/2022, 2:27 PMTom Klein
07/10/2022, 3:02 PMLocalDaskExecutor
does, for example)
it’s even described as “your” (prefect’s) “preferred” way of running:
https://stories.dask.org/en/latest/prefect-workflows.html
Our preferred deployment of Prefect Flows uses dask-kubernetes to spin up a short-lived Dask Cluster in Kubernetes.
so it’s not like i’m inventing the wheel here, but still feels a bit like itDask Gateway
- is that the “best practice” of how to create clusters on the fly?
https://docs.prefect.io/core/advanced_tutorials/dask-cluster.html
although, it’s not clear from this doc when exactly is the cluster created if it’s just being run from outside of the flow scope?Anna Geller
07/10/2022, 4:08 PMTom Klein
07/10/2022, 4:13 PM[2022-07-10 19:08:20+0300] INFO - prefect.FlowRunner | Beginning Flow run for 'lead_enrich_partial'
[2022-07-10 19:08:20+0300] INFO - prefect.DaskExecutor | Creating a new Dask cluster with `flows.partial_enrich.<lambda>`...
distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO - Scheduler at: <tcp://192.168.0.189:62569>
distributed.scheduler - INFO - dashboard at: :8787
distributed.core - INFO - Event loop was unresponsive in Scheduler for 200.43s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
[2022-07-10 19:11:40+0300] ERROR - prefect.FlowRunner | Unexpected error: ConnectionError('kubectl port forward failed')
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/prefect/engine/flow_runner.py", line 442, in get_flow_run_state
with self.check_for_cancellation(), executor.start():
File "/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 119, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.9/site-packages/prefect/executors/dask.py", line 238, in start
with self.cluster_class(**self.cluster_kwargs) as cluster:
File "/Users/klay/dev/honeybook/hb-prefect/flows/partial_enrich.py", line 232, in <lambda>
cluster_class=lambda: KubeCluster(make_pod_spec(image="<http://887300609994.dkr.ecr.us-east-1.amazonaws.com/hb-ds/lead-enrichment:2022|887300609994.dkr.ecr.us-east-1.amazonaws.com/hb-ds/lead-enrichment:2022>"),
File "/usr/local/lib/python3.9/site-packages/dask_kubernetes/classic/kubecluster.py", line 503, in __init__
super().__init__(**self.kwargs)
File "/usr/local/lib/python3.9/site-packages/distributed/deploy/spec.py", line 260, in __init__
self.sync(self._start)
File "/usr/local/lib/python3.9/site-packages/distributed/utils.py", line 310, in sync
return sync(
File "/usr/local/lib/python3.9/site-packages/distributed/utils.py", line 364, in sync
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.9/site-packages/distributed/utils.py", line 349, in f
result[0] = yield future
File "/usr/local/lib/python3.9/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/usr/local/lib/python3.9/site-packages/dask_kubernetes/classic/kubecluster.py", line 647, in _start
self.forwarded_dashboard_port = await port_forward_dashboard(
File "/usr/local/lib/python3.9/site-packages/dask_kubernetes/common/networking.py", line 110, in port_forward_dashboard
port = await port_forward_service(service_name, namespace, 8787)
File "/usr/local/lib/python3.9/site-packages/dask_kubernetes/common/networking.py", line 94, in port_forward_service
raise ConnectionError("kubectl port forward failed")
ConnectionError: kubectl port forward failed
[2022-07-10 19:11:40+0300] ERROR - prefect.lead_enrich_partial | Unexpected error occured in FlowRunner: ConnectionError('kubectl port forward failed')
ERROR:prefect.FlowRunner:Unexpected error: ConnectionError('kubectl port forward failed')
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/prefect/engine/flow_runner.py", line 442, in get_flow_run_state
with self.check_for_cancellation(), executor.start():
File "/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 119, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.9/site-packages/prefect/executors/dask.py", line 238, in start
with self.cluster_class(**self.cluster_kwargs) as cluster:
File "/Users/klay/dev/honeybook/hb-prefect/flows/partial_enrich.py", line 232, in <lambda>
cluster_class=lambda: KubeCluster(make_pod_spec(image="<http://887300609994.dkr.ecr.us-east-1.amazonaws.com/hb-ds/lead-enrichment:2022|887300609994.dkr.ecr.us-east-1.amazonaws.com/hb-ds/lead-enrichment:2022>"),
File "/usr/local/lib/python3.9/site-packages/dask_kubernetes/classic/kubecluster.py", line 503, in __init__
super().__init__(**self.kwargs)
File "/usr/local/lib/python3.9/site-packages/distributed/deploy/spec.py", line 260, in __init__
self.sync(self._start)
File "/usr/local/lib/python3.9/site-packages/distributed/utils.py", line 310, in sync
return sync(
File "/usr/local/lib/python3.9/site-packages/distributed/utils.py", line 364, in sync
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.9/site-packages/distributed/utils.py", line 349, in f
result[0] = yield future
File "/usr/local/lib/python3.9/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/usr/local/lib/python3.9/site-packages/dask_kubernetes/classic/kubecluster.py", line 647, in _start
self.forwarded_dashboard_port = await port_forward_dashboard(
File "/usr/local/lib/python3.9/site-packages/dask_kubernetes/common/networking.py", line 110, in port_forward_dashboard
port = await port_forward_service(service_name, namespace, 8787)
File "/usr/local/lib/python3.9/site-packages/dask_kubernetes/common/networking.py", line 94, in port_forward_service
raise ConnectionError("kubectl port forward failed")
ConnectionError: kubectl port forward failed
ERROR:prefect.lead_enrich_partial:Unexpected error occured in FlowRunner: ConnectionError('kubectl port forward failed')
distributed.scheduler - INFO - Scheduler closing...
distributed.scheduler - INFO - Scheduler closing all comms
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x122830970>
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x12284b310>
however, i don’t think the flow was actually being run on (my local) Kubernetes despite the Kubernetes run config?Anna Geller
07/10/2022, 4:14 PMTom Klein
07/10/2022, 4:41 PMyaml
into the registered flow (stored on S3) ? or does it need to be available on the image?
i’m guessing the latter?
wait, you’re not even using the yaml
in your flow though?annageller/prefect-dask-k8s:latest
can you share the Dockerfile for it?<http://ghcr.io/dask/dask|ghcr.io/dask/dask>
it still failed, as long as the Prefect flow ran on our custom image (despite it having kubectl
, dask-kuberentes
, dask
, etc.)
so it seems like a missing dependency on the “parent” (prefect flow / k8s job) rather than the “child” (dask scheduler or worker)FROM node:12-alpine
WORKDIR /usr/src/app
RUN apk update
RUN apk add bash curl py3-pip
RUN apk add --no-cache build-base libffi-dev openssl-dev g++ python3-dev
RUN pip3 install --upgrade pip setuptools wheel
RUN pip3 install numpy pandas
RUN pip3 install snowflake-connector-python --no-use-pep517
RUN pip3 install prefect[github,aws,kubernetes,snowflake,dask] --use-deprecated=legacy-resolver
RUN pip3 install dask
RUN curl -L "<https://dl.k8s.io/release/$(curl> -L -s <https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl>" -o /usr/local/bin/kubectl
RUN chmod +x /usr/local/bin/kubectl
Anna Geller
07/10/2022, 5:20 PMTom Klein
07/10/2022, 5:23 PMenv={"EXTRA_PIP_PACKAGES": "dask-kubernetes"}
but our custom image has the same thing, which makes it a phantom problem….
either some sort of version conflict/mismatch or an issue with the fact we use Alpine linuxAnna Geller
07/10/2022, 6:32 PMTom Klein
07/10/2022, 6:39 PMAnna Geller
07/10/2022, 6:40 PMTom Klein
07/10/2022, 6:40 PMLocalDaskExecutor
for example it works fantastic (ally?)DaskExecutor
not work correctly but everything else work fine (and even with that, it’s not that it fails on like a missing package, it runs — it just strangely fails on something, tries to fallback on kubectl
, and fails that too — probably as a completely irrelevant side-effect that hides the true issue)prefect
(and that it needs to be in the PATH) . that’s itAnna Geller
07/10/2022, 6:46 PMTom Klein
07/10/2022, 6:48 PM