Hi there, i’m having some issues with flow deployments, with an S3 block. My prefect agent pickups the flow run, starts the flow run but fails to get the flow

10:21:24.290 | INFO    | prefect.agent - Submitting flow run 'cfc4f262-4f05-4685-882e-364192297107'
10:21:24.474 | INFO    | prefect.infrastructure.process - Opening process 'blond-mammoth'...
10:21:24.482 | INFO    | prefect.agent - Completed submission of flow run 'cfc4f262-4f05-4685-882e-364192297107'
10:21:27.334 | ERROR   | Flow run 'blond-mammoth' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/opt/micromamba/envs/main/lib/python3.8/site-packages/prefect/engine.py", line 247, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/opt/micromamba/envs/main/lib/python3.8/site-packages/prefect/client.py", line 104, in with_injected_client
    return await fn(*args, **kwargs)
  File "/opt/micromamba/envs/main/lib/python3.8/site-packages/prefect/deployments.py", line 47, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=None, local_path=".")
  File "/opt/micromamba/envs/main/lib/python3.8/site-packages/prefect/filesystems.py", line 373, in get_directory
    return await self.filesystem.get_directory(
  File "/opt/micromamba/envs/main/lib/python3.8/site-packages/prefect/filesystems.py", line 251, in get_directory
    return self.filesystem.get(from_path, local_path, recursive=True)
  File "/opt/micromamba/envs/main/lib/python3.8/site-packages/fsspec/spec.py", line 801, in get
    self.get_file(rpath, lpath, **kwargs)
  File "/opt/micromamba/envs/main/lib/python3.8/site-packages/fsspec/spec.py", line 769, in get_file
    outfile = open(lpath, "wb")
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp3er_ugnvprefect/S3-BUCKET-NAME/alpha/flow.py'
10:21:27.727 | INFO    | prefect.infrastructure.process - Process 'blond-mammoth' exited cleanly.
... 
$ aws s3 ls <s3://S3-BUCKET-NAME/alpha/>
2022-08-06 10:20:49       6473 flow.py
2022-08-06 10:20:49       3204 example_flow-manifest.json

Created via

# prefect deployment build ./flow.py:example_flow --name example-flow-alpha --tag alpha --storage-block s3/S3-BUCKET-NAME
# prefect deployment apply example-flow-alpha.yaml

Had mixed luck reading and searching similar posts with this error message

this might be helpful https://discourse.prefect.io/t/how-to-deploy-prefect-2-0-flows-to-aws/1252

Thanks Anna, I’ll try that again – but something tells me there is another issue with my setup (probably sth very silly, as usual)

soon, you'll be able to deploy multiple flows using a single command

Hey guys, I'm also having a similar issue and have tried rerunning the above guide without success. What's interesting is that it works fine when I run the agent on Windows (sources flow from S3 no problems) but as soon as I go and run it on Linux I get the FileNotFoundError. Tried running under different user accounts, different python environments, and different versions of Linux (Centos, Ubuntu, Debian) and can't seem to get it to work. Any ideas for what I might be doing wrong?

many thanks for the update Anna, we’ll keep waiting 😉 The parametrized flow deployments are a MASSIVE step forward

Nice 👍

Just for anyone else who finds themselves stuck at this point and ends up here. In my case it seemed to have something to do with having boto3 and s3fs installed together. Remove boto3 and it works fine. Maybe some kind of old common package dependency or something

nice work, thx for the update!