Kevin Grismore
08/05/2022, 9:18 PM- project
└── flows
└── flow1.py
└── flow2.py
└── util
└── util.py
if I do some/dir/project> prefect deployment build flows/flow1.py:flow_func -n my-flow -ib kubernetes-job/my-job -sb gcs/my-bucket -t k8s
everything in src ends up in my bucket as expected, but when I run the flow I get:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/flows/flow1.py'
Anna Geller
08/05/2022, 11:26 PMCorris Randall
08/06/2022, 12:44 AMKevin Grismore
08/06/2022, 2:23 AM- project
└── flow1.py
└── flow2.py
└── util
└── __init__.py
└── util.py
ModuleNotFoundError: No module named 'util'
, or the same error for anything that's in a folder as part of what gets packaged with my flow in the gcs bucket, despite including init
files and everything running fine locally and on a local agentdeployment build
command on windows. when all the files are sent to my gcs bucket via shutil.copytree()
, all the nested directories get flattened in the bucket because it doesn't recognize the \
as a folder separator. Consequently, all the files get downloaded to the target execution environment as a flat structure too.
flow1.py
flow2.py
util
└── __init__.py
└── util.py
becomes
flow1.py
flow2.py
util\__init__.py
util\util.py
which... well of course python isn't going to find the modules it's trying to importshutil.copytree()
- it uses os.scandir()
. Even if you were to posix-ify the the base path, all the children will still have windows-style \
separators:
>>> path = Path('C:\Windows').as_posix()
>>> itr = os.scandir(path)
>>> [print(path.path) for path in itr]
C:/Windows\addins
C:/Windows\appcompat
C:/Windows\apppatch
C:/Windows\AppReadiness
Anna Geller
08/06/2022, 11:57 AM/
and \
separators"Marvin
08/06/2022, 11:58 AM