https://prefect.io logo
Powered by
Title
k

Kevin Grismore

08/05/2022, 9:18 PM
having trouble running my gcs-stored flows on kubernetes. I feel like it probably has something to do with how my project is structured: 
- project
        └── flows
            └── flow1.py
            └── flow2.py
        └── util
            └── util.py
if I do 
some/dir/project> prefect deployment build flows/flow1.py:flow_func -n my-flow -ib kubernetes-job/my-job -sb gcs/my-bucket -t k8s
everything in src ends up in my bucket as expected, but when I run the flow I get: 
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/flows/flow1.py'
a

Anna Geller

08/05/2022, 11:26 PM
A recipe for baking flow code into a docker image is on our radar and should be released within the next 2 weeks or so
:thank-you: 1
c

Corris Randall

08/06/2022, 12:44 AM
oooh, how serendipitous, this will be useful for me too!
k

Kevin Grismore

08/06/2022, 2:23 AM
so, I got around this by changing my project structure. 
- project
        └── flow1.py
        └── flow2.py
        └── util
            └── __init__.py
            └── util.py
but I still get 
ModuleNotFoundError: No module named 'util'
, or the same error for anything that's in a folder as part of what gets packaged with my flow in the gcs bucket, despite including 
init
files and everything running fine locally and on a local agent
actually, wait... I think I might know the issue. will come back to this if I'm right. I do this a lot lol
okay, soooooo I think it's because I'm running the 
deployment build
command on windows. when all the files are sent to my gcs bucket via 
shutil.copytree()
, all the nested directories get flattened in the bucket because it doesn't recognize the 
\
as a folder separator. Consequently, all the files get downloaded to the target execution environment as a flat structure too. 
flow1.py
flow2.py
util
    └── __init__.py
    └── util.py
becomes 
flow1.py
flow2.py
util\__init__.py
util\util.py
which... well of course python isn't going to find the modules it's trying to import
👀 1
this behavior looks like it's built directly in to 
shutil.copytree()
- it uses 
os.scandir()
. Even if you were to posix-ify the the base path, all the children will still have windows-style 
\
separators: 
>>> path = Path('C:\Windows').as_posix()
>>> itr = os.scandir(path)
>>> [print(path.path) for path in itr]
C:/Windows\addins
C:/Windows\appcompat
C:/Windows\apppatch
C:/Windows\AppReadiness
a

Anna Geller

08/06/2022, 11:57 AM
thanks for this thorough investigation. Let me open an issue @Marvin open "Deployments built on Windows and uploaded to GCS cause runtime errors due to 
/
and 
\
separators"
m

Marvin

08/06/2022, 11:58 AM
https://github.com/PrefectHQ/prefect/issues/6276
17 Views
#prefect-communityJoin Slack
Powered by