I’m having an issue with running flows on kubernetes (minikube). I have a server and an agent runni...

Alvaro Durán Tovar

04/21/2023, 7:44 PM

I’m having an issue with running flows on kubernetes (minikube). I have a server and an agent running in minikube. I’m using gcs as storage and kubernetes job as infrastructure. I have a file in my repo called

flows/api_flow.py

and the entrypoint is

./flows/api_flow.py:call_api

. The error comes when triggering the flow to run it on kubernetes. Failes with this error:

Copy code

prefect.exceptions.ScriptError: Script at 'flows/api_flow.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')

✅ 1

Alvaro Durán Tovar

04/21/2023, 7:45 PM

The gcs block is configured using this bucket path: “<something/flows>”

Alvaro Durán Tovar

04/21/2023, 7:47 PM

I can make it work by changing the bucket path to

<something>

but then it spams my bucket. The errors comes when adding a path to the bucket path, any idea? It’s not able to run the flow when it’s stored this way

<something>/flows/flows/api_flow.py

and the bucket path configured as

<something/flows>

Ryan Peden

04/21/2023, 8:59 PM

Someone else had a similar issue with Azure storage, which, like the GCP storage block, uses

fsspec

and seems to have some issues when using the two newest release of `fsspec`: https://github.com/PrefectHQ/prefect/issues/9262 So, intentionally installing

fsspec==2023.1.0

in your container image (instead of the newer

2023.3.0

2023.4.0

versions) might help. Also, someone else here on Slack reported earlier today that setting

path=""

in their deployment solved the issue for them. I don't know if that will work on your case, but I figure it is worth mentioning.

🙌 1

5 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.