https://prefect.io
Join Slack
having trouble running my gcs-stored flows on kubernetes. I feel like it probably has something to d...
# ask-community
k
having trouble running my gcs-stored flows on kubernetes. I feel like it probably has something to do with how my project is structured:
Copy code
- project
        └── flows
            └── flow1.py
            └── flow2.py
        └── util
            └── util.py
if I do 
some/dir/project> prefect deployment build flows/flow1.py:flow_func -n my-flow -ib kubernetes-job/my-job -sb gcs/my-bucket -t k8s
everything in src ends up in my bucket as expected, but when I run the flow I get:
Copy code
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/flows/flow1.py'
a
A recipe for baking flow code into a docker image is on our radar and should be released within the next 2 weeks or so
🙏 1
c
oooh, how serendipitous, this will be useful for me too!
k
so, I got around this by changing my project structure.
Copy code
- project
        └── flow1.py
        └── flow2.py
        └── util
            └── __init__.py
            └── util.py
but I still get 
ModuleNotFoundError: No module named 'util'
, or the same error for anything that's in a folder as part of what gets packaged with my flow in the gcs bucket, despite including 
init
files and everything running fine locally and on a local agent
actually, wait... I think I might know the issue. will come back to this if I'm right. I do this a lot lol
okay, soooooo I think it's because I'm running the 
deployment build
command on windows. when all the files are sent to my gcs bucket via 
shutil.copytree()
, all the nested directories get flattened in the bucket because it doesn't recognize the 
\
as a folder separator. Consequently, all the files get downloaded to the target execution environment as a flat structure too.
Copy code
flow1.py
flow2.py
util
    └── __init__.py
    └── util.py
becomes
Copy code
flow1.py
flow2.py
util\__init__.py
util\util.py
which... well of course python isn't going to find the modules it's trying to import
👀 1
this behavior looks like it's built directly in to 
shutil.copytree()
- it uses 
os.scandir()
. Even if you were to posix-ify the the base path, all the children will still have windows-style 
\
separators:
Copy code
>>> path = Path('C:\Windows').as_posix()
>>> itr = os.scandir(path)
>>> [print(path.path) for path in itr]
C:/Windows\addins
C:/Windows\appcompat
C:/Windows\apppatch
C:/Windows\AppReadiness
a
thanks for this thorough investigation. Let me open an issue @Marvin open "Deployments built on Windows and uploaded to GCS cause runtime errors due to 
/
and 
\
separators"
m
91 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by