Hey guys! I am having an issue with running flows as KubernetesJob together with prefect UI v2.0.2 in my GCP Kubernetes. let me describe all the steps. I guess this may be useful for other guys in community, as prefect docs/ho-tos are becoming quickly outdated due to rapid development: • save the output of

prefect kubernetes manifest orion

to

k8s_deployment.yaml

(please add the ability to set non-default k8s namespace) • change PREFECT_API_URL from http://orion:4200/api to http://0.0.0.0:4200/api •

kubectl apply -f k8s_deployment.yaml

(it already has command: [“prefect”, “orion”, “start”, “--host”, “0.0.0.0”, “--log-level”, “WARNING”] for API container) • check the GCP admin. The agent/ui/api seems to be healthy. •

kubectl port-forward deployment/orion 4200:4200

• check the UI is healthy via http://127.0.0.1:4200/. •

prefect config set PREFECT_API_URL=<http://0.0.0.0:4200/api>

• register GCS block via admin panel. name=prefect. path: gs://{my-path}. (prefect block register -f gcs_block.py didn’t work for me for some reason) •

prefect deployment build ../flows/backtest.py:backtest_flow --name dev --tag dev --infra kubernetes-job --storage-block gcs/prefect

(should i run it from root of the project?) • checked my GCS bucket. The files did appear in the local folder and GCS. •

prefect deployment apply backtest_flow-deployment.yaml

• check UI http://127.0.0.1:4200/deployments. deployment did appear • create

kubernetes

queue manually. The name seems to be hardcoded according to the error logs in GCP agent. This step fixes the error. •

prefect deployment run Backtest/dev

• the flow run was created. But it is stuck in pending. • The appropriate KubernetesJob was created. But it is stuck in error state with the error All connection attempts failed. and ConnectionRefusedError: [Errno 111] Connect call failed (‘0.0.0.0’, 4200). Please help! Thank you in advance :)

Can you perhaps try setting localhost instead of 0.0.0.0?

prefect config set PREFECT_API_URL=<http://localhost:4200/api>

@Anna Geller I really appreciate your support and the software you build! I’ll try your idea tomorrow in the morning.

(please add the ability to set non-default k8s namespace)

You can manually edit the .yaml to use a non-default namespace, wherever you have a block (where you find apiVersion and kind), either change or add metadata/namespace:

apiVersion: <...>
kind: <...>
metadata:
  namespace: prefect

@Dominik Wagner good point, i did it. But i don’t know how to set namespace for KubernetesJob. And it is quite strange to have orion UI in the custom namespace and jobs in the other one.

Ah I created a KubernetesJob infrastructure block, and there you can specify the namespace

exactly as Dominik said, @Mike Kovetsky you can fully customize every Kubernetes object to run in the namespace you want using metadata field

great! Some urgent stuff came up, so I am not sure I will have time to try this today. But i’ll surely do it on Monday 🙂

sorry, never deployed 2.0 to GCP so don't know how to help