https://prefect.io logo
Title
j

Jan Domanski

08/06/2022, 10:24 AM
Hi there, i’m having some issues with flow deployments, with an S3 block. My prefect agent pickups the flow run, starts the flow run but fails to get the flow
10:21:24.290 | INFO    | prefect.agent - Submitting flow run 'cfc4f262-4f05-4685-882e-364192297107'
10:21:24.474 | INFO    | prefect.infrastructure.process - Opening process 'blond-mammoth'...
10:21:24.482 | INFO    | prefect.agent - Completed submission of flow run 'cfc4f262-4f05-4685-882e-364192297107'
10:21:27.334 | ERROR   | Flow run 'blond-mammoth' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/opt/micromamba/envs/main/lib/python3.8/site-packages/prefect/engine.py", line 247, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/opt/micromamba/envs/main/lib/python3.8/site-packages/prefect/client.py", line 104, in with_injected_client
    return await fn(*args, **kwargs)
  File "/opt/micromamba/envs/main/lib/python3.8/site-packages/prefect/deployments.py", line 47, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=None, local_path=".")
  File "/opt/micromamba/envs/main/lib/python3.8/site-packages/prefect/filesystems.py", line 373, in get_directory
    return await self.filesystem.get_directory(
  File "/opt/micromamba/envs/main/lib/python3.8/site-packages/prefect/filesystems.py", line 251, in get_directory
    return self.filesystem.get(from_path, local_path, recursive=True)
  File "/opt/micromamba/envs/main/lib/python3.8/site-packages/fsspec/spec.py", line 801, in get
    self.get_file(rpath, lpath, **kwargs)
  File "/opt/micromamba/envs/main/lib/python3.8/site-packages/fsspec/spec.py", line 769, in get_file
    outfile = open(lpath, "wb")
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp3er_ugnvprefect/S3-BUCKET-NAME/alpha/flow.py'
10:21:27.727 | INFO    | prefect.infrastructure.process - Process 'blond-mammoth' exited cleanly.
... 
$ aws s3 ls <s3://S3-BUCKET-NAME/alpha/>
2022-08-06 10:20:49       6473 flow.py
2022-08-06 10:20:49       3204 example_flow-manifest.json
Created via
# prefect deployment build ./flow.py:example_flow --name example-flow-alpha --tag alpha --storage-block s3/S3-BUCKET-NAME
# prefect deployment apply example-flow-alpha.yaml
Had mixed luck reading and searching similar posts with this error message
1
Gotta say the docs are pretty spares too https://docs.prefect.io/ui/cloud-getting-started/#configure-storage None of these local flows have a chance of running on a remote agent, right?
a

Anna Geller

08/06/2022, 11:37 AM
j

Jan Domanski

08/06/2022, 12:08 PM
Thanks Anna, I’ll try that again – but something tells me there is another issue with my setup (probably sth very silly, as usual)
🙌 1
Brilliant this worked – thank you! What’s your recommendation for how to do this for multiple flow? Multiple buckets? Single bucket but a single prefix per flow? How to put this together with blocks?
Thinking of a very standard scenario, with N flows, each with a prod/staging/dev envs
a

Anna Geller

08/07/2022, 9:28 PM
soon, you'll be able to deploy multiple flows using a single command
j

Jack Sinclair

08/08/2022, 12:55 AM
Hey guys, I'm also having a similar issue and have tried rerunning the above guide without success. What's interesting is that it works fine when I run the agent on Windows (sources flow from S3 no problems) but as soon as I go and run it on Linux I get the FileNotFoundError. Tried running under different user accounts, different python environments, and different versions of Linux (Centos, Ubuntu, Debian) and can't seem to get it to work. Any ideas for what I might be doing wrong?
j

Jan Domanski

08/08/2022, 3:31 PM
many thanks for the update Anna, we’ll keep waiting 😉 The parametrized flow deployments are a MASSIVE step forward
a

Anna Geller

08/08/2022, 7:38 PM
Nice 👍
j

Jack Sinclair

09/08/2022, 12:51 AM
Just for anyone else who finds themselves stuck at this point and ends up here. In my case it seemed to have something to do with having boto3 and s3fs installed together. Remove boto3 and it works fine. Maybe some kind of old common package dependency or something
🙌 1
a

Anna Geller

09/08/2022, 9:24 AM
nice work, thx for the update!