Hi, I'm trying to work through the Prefect Orion tutorial for running flows in Kubernetes (https://orion-docs.prefect.io/tutorials/kubernetes-flow-runner/) and I keep running into the same problem. When I run the deployment,

prefect deployment run test-flow/test-deployment

, the UI shows that the flow run is Pending. When I look at the logs in Kubernetes with

kubectl logs -l job-name=voracious-antelope2b89g

, I see a FileNotFound stack trace Thank you for your help!

Hi @James Harr, I might have time to try to replicate this later but if I don’t, I will surely ask some of the engineers on Monday. Would appreciate it if you could move the traceback to this thread when you get a chance to keep the main channel cleaner

No problem, thank you for taking a look!

async with await anyio.open_file(storage_path, mode="rb") as fp:
  File "/usr/local/lib/python3.9/site-packages/anyio/_core/_fileio.py", line 156, in open_file
    fp = await to_thread.run_sync(open, file, mode, buffering, encoding, errors, newline,
  File "/usr/local/lib/python3.9/site-packages/anyio/to_thread.py", line 28, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(func, *args, cancellable=cancellable,
  File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 818, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 754, in run
    result = context.run(func, *args)
FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/0q/dynsm9h90vjb8py9gz9m0q8c0000gn/T/prefect/f000f05a-ab00-4b66-b673-aed9dfb57bef'

When I look at the Deployments tab in the UI, I can see my deployment with some summary text that says:

{
  "data": "\"/var/folders/0q/dynsm9h90vjb8py9gz9m0q8c0000gn/T/prefect/f000f05a-ab00-4b66-b673-aed9dfb57bef\"",
  "block_id": null
}

The value for

data

is the same location that the pod logs say cannot be found. My guess is that my

k8s_flow.py

file is not being copied into the container that the pod runs. Also, when I create the deployment I get a warning:

.../prefect/client.py:1227: UserWarning: No default storage has been set on the server. Using temporary local storage for results.

Is this part of the problem also? It's using my local machine to store the deployment info but then when deploying into Kubernetes it doesn't have access to that file?

Thank you!

Basically yeah you need to use non-local storage for the deployment to be retrievable from K8s