Tom Klein
07/10/2022, 9:54 AMTom Klein
07/10/2022, 9:55 AMUnexpected error: RuntimeError('Missing dependency kubectl. Please install kubectl following the instructions for your OS. ')
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/prefect/engine/flow_runner.py", line 442, in get_flow_run_state
with self.check_for_cancellation(), executor.start():
File "/usr/lib/python3.9/contextlib.py", line 119, in __enter__
return next(self.gen)
File "/usr/lib/python3.9/site-packages/prefect/executors/dask.py", line 243, in start
with self.cluster_class(**self.cluster_kwargs) as cluster:
File "<string>", line 215, in <lambda>
File "/usr/lib/python3.9/site-packages/dask_kubernetes/classic/kubecluster.py", line 503, in __init__
super().__init__(**self.kwargs)
File "/usr/lib/python3.9/site-packages/distributed/deploy/spec.py", line 264, in __init__
self.sync(self._start)
File "/usr/lib/python3.9/site-packages/distributed/utils.py", line 336, in sync
return sync(
File "/usr/lib/python3.9/site-packages/distributed/utils.py", line 403, in sync
raise exc.with_traceback(tb)
File "/usr/lib/python3.9/site-packages/distributed/utils.py", line 376, in f
result = yield future
File "/usr/lib/python3.9/site-packages/tornado/gen.py", line 769, in run
value = future.result()
File "/usr/lib/python3.9/site-packages/dask_kubernetes/classic/kubecluster.py", line 647, in _start
self.forwarded_dashboard_port = await port_forward_dashboard(
File "/usr/lib/python3.9/site-packages/dask_kubernetes/common/networking.py", line 110, in port_forward_dashboard
port = await port_forward_service(service_name, namespace, 8787)
File "/usr/lib/python3.9/site-packages/dask_kubernetes/common/networking.py", line 75, in port_forward_service
check_dependency("kubectl")
File "/usr/lib/python3.9/site-packages/dask_kubernetes/common/utils.py", line 38, in check_dependency
raise RuntimeError(
RuntimeError: Missing dependency kubectl. Please install kubectl following the instructions for your OS.
Tom Klein
07/10/2022, 9:56 AMkubernetes
as an extra for the prefect pip installation, not enough?Tom Klein
07/10/2022, 9:56 AMwith Flow(
FLOW_NAME,
result=s3_result,
executor=DaskExecutor(
cluster_class=lambda: KubeCluster(make_pod_spec(image=prefect.context.image)),
adapt_kwargs={"minimum": 2, "maximum": 2},
),
run_config=KubernetesRun(
image="xyz_my_image",
memory_request="32Gi",
env=my_env,
),
) as flow:
Tom Klein
07/10/2022, 9:59 AMTom Klein
07/10/2022, 10:02 AMcurl -LO <https://dl.k8s.io/release/v1.21.2/bin/linux/amd64/kubectl> &&\
install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl &&\
lines to our docker image?
i thought that Prefect comes with the necessary Dask/Kubernetes dependencies?Anna Geller
Tom Klein
07/10/2022, 10:31 AMdask-kubernetes
packaged along at least the Kubernetes
extra (and that the latter would install kubectl if necessary)Anna Geller
Tom Klein
07/10/2022, 10:38 AMTom Klein
07/10/2022, 10:47 AMTom Klein
07/10/2022, 11:06 AMkubectl
was successfully installed but seems like it’s missing config:
Step 12/23 : RUN kubectl version
---> Running in 6f3b72feadad
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
The connection to the server localhost:8080 was refused - did you specify the right host or port?
Client Version: <http://version.Info|version.Info>{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:22:29Z", GoVersion:"go1.18.3", Compiler:"gc", Plat
form:"linux/amd64"}
Kustomize Version: v4.5.4
[SYSTEM]
Message Failed to build image: honeybook/ds-lead-enrichment:2022
Caused by Container for step title: Building Docker Image, step type: build, operation: Building image.
this is what i don’t get - do we need to handle these low-level configs as part of working with prefect? i thought prefect takes care of the interaction with the kubernetes cluster for us (e.g. we didn’t have this issue when running Kubernetes tasks)Tom Klein
07/10/2022, 11:28 AMAnna Geller
Anna Geller
Anna Geller
Tom Klein
07/10/2022, 12:00 PMaws cli
because it supposedly gives the ability to fetch/create a config file for kubectl
automatically - and now it’s failing because despite our Prefect deployments (e.g. the agent, the jobs) having a k8s servicerole with full AWS permissions, it still tells me:
Removing intermediate container b69c69743ae3
---> 4c2d52aa2e3a
Step 13/24 : RUN aws eks update-kubeconfig --region us-east-1 --name prefect
---> Running in deb16c5fa885
Unable to locate credentials. You can configure credentials by running "aws configure".
i just hoped that we wouldn’t have to get to these kind of low-level configurations when using Prefect and that once we set it up it simply integrates with what we need (given that it has the permissions)
i thought it’s mostly a matter of giving it an executor with an integer representing the number of workers and an imageTom Klein
07/10/2022, 12:01 PMRunNamespacedJob
) i didn’t have to do any of that stuffAnna Geller
Tom Klein
07/10/2022, 12:02 PMAnna Geller
Tom Klein
07/10/2022, 12:05 PMTom Klein
07/10/2022, 12:12 PMENTRYPOINT
for docker?Anna Geller
Anna Geller
Tom Klein
07/10/2022, 12:19 PMAnna Geller
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1
Tom Klein
07/10/2022, 12:20 PMkubectl
Tom Klein
07/10/2022, 12:30 PMprefect.context.image
is a legitimate reference? cause i don’t see it in the docs:
https://docs.prefect.io/api/latest/utilities/context.html#context-2Tom Klein
07/10/2022, 12:46 PMvalue = future.result()
File "/usr/lib/python3.9/site-packages/dask_kubernetes/classic/kubecluster.py", line 647, in _start
self.forwarded_dashboard_port = await port_forward_dashboard(
File "/usr/lib/python3.9/site-packages/dask_kubernetes/common/networking.py", line 110, in port_forward_dashboard
port = await port_forward_service(service_name, namespace, 8787)
File "/usr/lib/python3.9/site-packages/dask_kubernetes/common/networking.py", line 94, in port_forward_service
raise ConnectionError("kubectl port forward failed")
ConnectionError: kubectl port forward failed
i understand this is more of a dask-kubernetes issue than a prefect issue — but any chance you have any idea what’s happening?Anna Geller
Anna Geller
Tom Klein
07/10/2022, 12:59 PMno-dahsboard
would help? the pod is created with everything — i suspect the issue might be more fundamentalTom Klein
07/10/2022, 1:00 PMTom Klein
07/10/2022, 1:05 PMOliver Mannion
07/10/2022, 1:33 PMTom Klein
07/10/2022, 1:34 PMTom Klein
07/10/2022, 1:36 PMTom Klein
07/10/2022, 1:37 PMTom Klein
07/10/2022, 1:38 PMTom Klein
07/10/2022, 1:39 PMOliver Mannion
07/10/2022, 1:41 PMkubectl get service
?Tom Klein
07/10/2022, 1:41 PMTom Klein
07/10/2022, 1:42 PMkubectl
myself, only our devops have permissions for that, this ^^ is what they sent me to show that the pod was created)Tom Klein
07/10/2022, 1:42 PMClusterIP
?Anna Geller
Tom Klein
07/10/2022, 3:02 PMLocalDaskExecutor
does, for example)
it’s even described as “your” (prefect’s) “preferred” way of running:
https://stories.dask.org/en/latest/prefect-workflows.html
Our preferred deployment of Prefect Flows uses dask-kubernetes to spin up a short-lived Dask Cluster in Kubernetes.
so it’s not like i’m inventing the wheel here, but still feels a bit like itTom Klein
07/10/2022, 3:04 PMDask Gateway
- is that the “best practice” of how to create clusters on the fly?
https://docs.prefect.io/core/advanced_tutorials/dask-cluster.html
although, it’s not clear from this doc when exactly is the cluster created if it’s just being run from outside of the flow scope?Anna Geller
Anna Geller
Tom Klein
07/10/2022, 4:13 PM[2022-07-10 19:08:20+0300] INFO - prefect.FlowRunner | Beginning Flow run for 'lead_enrich_partial'
[2022-07-10 19:08:20+0300] INFO - prefect.DaskExecutor | Creating a new Dask cluster with `flows.partial_enrich.<lambda>`...
distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO - Scheduler at: <tcp://192.168.0.189:62569>
distributed.scheduler - INFO - dashboard at: :8787
distributed.core - INFO - Event loop was unresponsive in Scheduler for 200.43s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
[2022-07-10 19:11:40+0300] ERROR - prefect.FlowRunner | Unexpected error: ConnectionError('kubectl port forward failed')
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/prefect/engine/flow_runner.py", line 442, in get_flow_run_state
with self.check_for_cancellation(), executor.start():
File "/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 119, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.9/site-packages/prefect/executors/dask.py", line 238, in start
with self.cluster_class(**self.cluster_kwargs) as cluster:
File "/Users/klay/dev/honeybook/hb-prefect/flows/partial_enrich.py", line 232, in <lambda>
cluster_class=lambda: KubeCluster(make_pod_spec(image="<http://887300609994.dkr.ecr.us-east-1.amazonaws.com/hb-ds/lead-enrichment:2022|887300609994.dkr.ecr.us-east-1.amazonaws.com/hb-ds/lead-enrichment:2022>"),
File "/usr/local/lib/python3.9/site-packages/dask_kubernetes/classic/kubecluster.py", line 503, in __init__
super().__init__(**self.kwargs)
File "/usr/local/lib/python3.9/site-packages/distributed/deploy/spec.py", line 260, in __init__
self.sync(self._start)
File "/usr/local/lib/python3.9/site-packages/distributed/utils.py", line 310, in sync
return sync(
File "/usr/local/lib/python3.9/site-packages/distributed/utils.py", line 364, in sync
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.9/site-packages/distributed/utils.py", line 349, in f
result[0] = yield future
File "/usr/local/lib/python3.9/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/usr/local/lib/python3.9/site-packages/dask_kubernetes/classic/kubecluster.py", line 647, in _start
self.forwarded_dashboard_port = await port_forward_dashboard(
File "/usr/local/lib/python3.9/site-packages/dask_kubernetes/common/networking.py", line 110, in port_forward_dashboard
port = await port_forward_service(service_name, namespace, 8787)
File "/usr/local/lib/python3.9/site-packages/dask_kubernetes/common/networking.py", line 94, in port_forward_service
raise ConnectionError("kubectl port forward failed")
ConnectionError: kubectl port forward failed
[2022-07-10 19:11:40+0300] ERROR - prefect.lead_enrich_partial | Unexpected error occured in FlowRunner: ConnectionError('kubectl port forward failed')
ERROR:prefect.FlowRunner:Unexpected error: ConnectionError('kubectl port forward failed')
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/prefect/engine/flow_runner.py", line 442, in get_flow_run_state
with self.check_for_cancellation(), executor.start():
File "/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 119, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.9/site-packages/prefect/executors/dask.py", line 238, in start
with self.cluster_class(**self.cluster_kwargs) as cluster:
File "/Users/klay/dev/honeybook/hb-prefect/flows/partial_enrich.py", line 232, in <lambda>
cluster_class=lambda: KubeCluster(make_pod_spec(image="<http://887300609994.dkr.ecr.us-east-1.amazonaws.com/hb-ds/lead-enrichment:2022|887300609994.dkr.ecr.us-east-1.amazonaws.com/hb-ds/lead-enrichment:2022>"),
File "/usr/local/lib/python3.9/site-packages/dask_kubernetes/classic/kubecluster.py", line 503, in __init__
super().__init__(**self.kwargs)
File "/usr/local/lib/python3.9/site-packages/distributed/deploy/spec.py", line 260, in __init__
self.sync(self._start)
File "/usr/local/lib/python3.9/site-packages/distributed/utils.py", line 310, in sync
return sync(
File "/usr/local/lib/python3.9/site-packages/distributed/utils.py", line 364, in sync
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.9/site-packages/distributed/utils.py", line 349, in f
result[0] = yield future
File "/usr/local/lib/python3.9/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/usr/local/lib/python3.9/site-packages/dask_kubernetes/classic/kubecluster.py", line 647, in _start
self.forwarded_dashboard_port = await port_forward_dashboard(
File "/usr/local/lib/python3.9/site-packages/dask_kubernetes/common/networking.py", line 110, in port_forward_dashboard
port = await port_forward_service(service_name, namespace, 8787)
File "/usr/local/lib/python3.9/site-packages/dask_kubernetes/common/networking.py", line 94, in port_forward_service
raise ConnectionError("kubectl port forward failed")
ConnectionError: kubectl port forward failed
ERROR:prefect.lead_enrich_partial:Unexpected error occured in FlowRunner: ConnectionError('kubectl port forward failed')
distributed.scheduler - INFO - Scheduler closing...
distributed.scheduler - INFO - Scheduler closing all comms
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x122830970>
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x12284b310>
however, i don’t think the flow was actually being run on (my local) Kubernetes despite the Kubernetes run config?Anna Geller
Anna Geller
Anna Geller
Anna Geller
Tom Klein
07/10/2022, 4:41 PMTom Klein
07/10/2022, 4:43 PMyaml
into the registered flow (stored on S3) ? or does it need to be available on the image?
i’m guessing the latter?
wait, you’re not even using the yaml
in your flow though?Tom Klein
07/10/2022, 4:50 PMannageller/prefect-dask-k8s:latest
can you share the Dockerfile for it?Tom Klein
07/10/2022, 5:07 PM<http://ghcr.io/dask/dask|ghcr.io/dask/dask>
it still failed, as long as the Prefect flow ran on our custom image (despite it having kubectl
, dask-kuberentes
, dask
, etc.)
so it seems like a missing dependency on the “parent” (prefect flow / k8s job) rather than the “child” (dask scheduler or worker)Tom Klein
07/10/2022, 5:09 PMFROM node:12-alpine
WORKDIR /usr/src/app
RUN apk update
RUN apk add bash curl py3-pip
RUN apk add --no-cache build-base libffi-dev openssl-dev g++ python3-dev
RUN pip3 install --upgrade pip setuptools wheel
RUN pip3 install numpy pandas
RUN pip3 install snowflake-connector-python --no-use-pep517
RUN pip3 install prefect[github,aws,kubernetes,snowflake,dask] --use-deprecated=legacy-resolver
RUN pip3 install dask
RUN curl -L "<https://dl.k8s.io/release/$(curl> -L -s <https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl>" -o /usr/local/bin/kubectl
RUN chmod +x /usr/local/bin/kubectl
Anna Geller
Tom Klein
07/10/2022, 5:23 PMTom Klein
07/10/2022, 5:23 PMenv={"EXTRA_PIP_PACKAGES": "dask-kubernetes"}
but our custom image has the same thing, which makes it a phantom problem….
either some sort of version conflict/mismatch or an issue with the fact we use Alpine linuxTom Klein
07/10/2022, 5:26 PMAnna Geller
Tom Klein
07/10/2022, 6:39 PMAnna Geller
Tom Klein
07/10/2022, 6:40 PMLocalDaskExecutor
for example it works fantastic (ally?)Tom Klein
07/10/2022, 6:42 PMDaskExecutor
not work correctly but everything else work fine (and even with that, it’s not that it fails on like a missing package, it runs — it just strangely fails on something, tries to fallback on kubectl
, and fails that too — probably as a completely irrelevant side-effect that hides the true issue)Tom Klein
07/10/2022, 6:44 PMprefect
(and that it needs to be in the PATH) . that’s itTom Klein
07/10/2022, 6:44 PMTom Klein
07/10/2022, 6:45 PMAnna Geller
Anna Geller
Tom Klein
07/10/2022, 6:48 PMTom Klein
07/10/2022, 7:28 PM