https://prefect.io logo
Title
g

Gabriel Moran

04/10/2023, 1:40 PM
Good morning, I'm getting this error with Prefect 2 when deploying a flow from a python script via
Deployment.build_from_flow
in AWS ECS. Do you have any idea of what could be happening? I tried updating the deployment configuration in a lot of ways but it didn't help. The deployment is configured to store the flow in S3 and I think the permissions are correctly configured.
Traceback (most recent call last):
File "/app/flows/register_flows.py", line 88, in <module>
asyncio.run(register_flows())
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/app/flows/register_flows.py", line 71, in register_flows
deployment = await Deployment.build_from_flow(
File "/usr/local/lib/python3.8/site-packages/prefect/deployments.py", line 793, in build_from_flow
await deployment.upload_to_storage(ignore_file=ignore_file)
File "/usr/local/lib/python3.8/site-packages/prefect/deployments.py", line 638, in upload_to_storage
file_count = await self.storage.put_directory(
File "/usr/local/lib/python3.8/site-packages/prefect/filesystems.py", line 497, in put_directory
return await self.filesystem.put_directory(
File "/usr/local/lib/python3.8/site-packages/prefect/filesystems.py", line 352, in put_directory
included_files = filter_files(
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/filesystem.py", line 42, in filter_files
ignored_files = {p.path for p in spec.match_tree_entries(root)}
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/filesystem.py", line 42, in <setcomp>
ignored_files = {p.path for p in spec.match_tree_entries(root)}
File "/usr/local/lib/python3.8/site-packages/pathspec/pathspec.py", line 239, in match_tree_entries
yield from self.match_entries(entries)
File "/usr/local/lib/python3.8/site-packages/pathspec/pathspec.py", line 155, in match_entries
for entry in entries:
File "/usr/local/lib/python3.8/site-packages/pathspec/util.py", line 175, in iter_tree_entries
yield from _iter_tree_entries_next(os.path.abspath(root), '', {}, on_error, follow_links)
File "/usr/local/lib/python3.8/site-packages/pathspec/util.py", line 245, in _iter_tree_entries_next
yield from _iter_tree_entries_next(root_full, node_rel, memo, on_error, follow_links)
File "/usr/local/lib/python3.8/site-packages/pathspec/util.py", line 245, in _iter_tree_entries_next
yield from _iter_tree_entries_next(root_full, node_rel, memo, on_error, follow_links)
File "/usr/local/lib/python3.8/site-packages/pathspec/util.py", line 245, in _iter_tree_entries_next
yield from _iter_tree_entries_next(root_full, node_rel, memo, on_error, follow_links)
[Previous line repeated 3 more times]
File "/usr/local/lib/python3.8/site-packages/pathspec/util.py", line 214, in _iter_tree_entries_next
raise RecursionError(real_path=dir_real, first_path=memo[dir_real], second_path=dir_rel)
pathspec.util.RecursionError: ('/', '', 'proc/self/task/12/fd/12')
c

Chris White

04/10/2023, 3:55 PM
Issues like this are often caused by using Python to perform your deployment upload; I'm guessing that if you instead used the CLI interface this would get resolved. In addition, we are rolling out a new (hopefully more transparent!) way of managing your deployment configurations as described in this tutorial - would love some early feedback if you're willing to try it out and see if it resolves this issue!