https://prefect.io logo
#prefect-community
Title
# prefect-community
m

Mike Kovetsky

08/04/2022, 4:40 PM
Hey guys! I am having an issue with running flows as KubernetesJob together with prefect UI v2.0.2 in my GCP Kubernetes. let me describe all the steps. I guess this may be useful for other guys in community, as prefect docs/ho-tos are becoming quickly outdated due to rapid development: • save the output of
prefect kubernetes manifest orion
to
k8s_deployment.yaml
(please add the ability to set non-default k8s namespace) • change PREFECT_API_URL from http://orion:4200/api to http://0.0.0.0:4200/api
kubectl apply -f k8s_deployment.yaml
(it already has command: [“prefect”, “orion”, “start”, “--host”, “0.0.0.0”, “--log-level”, “WARNING”] for API container) • check the GCP admin. The agent/ui/api seems to be healthy. •
kubectl port-forward deployment/orion 4200:4200
• check the UI is healthy via http://127.0.0.1:4200/. •
prefect config set PREFECT_API_URL=<http://0.0.0.0:4200/api>
• register GCS block via admin panel. name=prefect. path: gs://{my-path}. (prefect block register -f gcs_block.py didn’t work for me for some reason) •
prefect deployment build ../flows/backtest.py:backtest_flow --name dev --tag dev --infra kubernetes-job --storage-block gcs/prefect
(should i run it from root of the project?) • checked my GCS bucket. The files did appear in the local folder and GCS. •
prefect deployment apply backtest_flow-deployment.yaml
• check UI http://127.0.0.1:4200/deployments. deployment did appear • create
kubernetes
queue manually. The name seems to be hardcoded according to the error logs in GCP agent. This step fixes the error. •
prefect deployment run Backtest/dev
• the flow run was created. But it is stuck in pending. • The appropriate KubernetesJob was created. But it is stuck in error state with the error All connection attempts failed. and ConnectionRefusedError: [Errno 111] Connect call failed (‘0.0.0.0’, 4200). Please help! Thank you in advance :)
a

Anna Geller

08/04/2022, 7:39 PM
Can you perhaps try setting localhost instead of 0.0.0.0?
Copy code
prefect config set PREFECT_API_URL=<http://localhost:4200/api>
m

Mike Kovetsky

08/04/2022, 10:31 PM
@Anna Geller I really appreciate your support and the software you build! I’ll try your idea tomorrow in the morning.
🙌 1
d

Dominik Wagner

08/05/2022, 7:12 AM
(please add the ability to set non-default k8s namespace)
You can manually edit the .yaml to use a non-default namespace, wherever you have a block (where you find apiVersion and kind), either change or add metadata/namespace:
Copy code
apiVersion: <...>
kind: <...>
metadata:
  namespace: prefect
🙏 1
m

Mike Kovetsky

08/05/2022, 7:27 AM
@Dominik Wagner good point, i did it. But i don’t know how to set namespace for KubernetesJob. And it is quite strange to have orion UI in the custom namespace and jobs in the other one.
d

Dominik Wagner

08/05/2022, 7:31 AM
Ah I created a KubernetesJob infrastructure block, and there you can specify the namespace
🙏 1
a

Anna Geller

08/05/2022, 11:16 AM
exactly as Dominik said, @Mike Kovetsky you can fully customize every Kubernetes object to run in the namespace you want using metadata field
m

Mike Kovetsky

08/05/2022, 11:36 AM
great! Some urgent stuff came up, so I am not sure I will have time to try this today. But i’ll surely do it on Monday 🙂
🙌 1
@Anna Geller unfortunately, the fix did not help. I think this is because the Kubernetes Job is running remotely in GCP and knows nothing about port forwarding and my local PC. So it cannot connect to Orion UI in the cluster. Should expose orion UI to the internet and use its public IP address for the api ?
Ah maybe it is a bad idea, as there is not authorization for the orion ui
the
Copy code
prefect config set PREFECT_API_URL=<http://localhost:4200/api>
may have been overwritten by the env variable value inside the deployment spec. I’ll redeploy with localhost and come back 🙂
no, it is the same 😞 OSError: [Errno 99] Cannot assign requested address ConnectionRefusedError: [Errno 111] Connect call failed (‘127.0.0.1’, 4200)
a

Anna Geller

08/05/2022, 7:52 PM
sorry, never deployed 2.0 to GCP so don't know how to help
7 Views