Hello wondering if anybody else is experiencing this issue A Prefect Community #prefect-aws

Hello, wondering if anybody else is experiencing t...

Nick Gohman

12/03/2023, 8:31 PM

Hello, wondering if anybody else is experiencing this issue. All of our flows in prefect have been failing since 6am CST this morning and we aren’t sure what’s going on. We store our deployments in AWS s3 and have been seeing errors, see thread for details.

Nick Gohman

12/03/2023, 8:31 PM

Copy code

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/engine.py", line 406, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/deployments/deployments.py", line 235, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=from_path, local_path=".")
  File "/usr/local/lib/python3.11/site-packages/prefect/filesystems.py", line 500, in get_directory
    return await self.filesystem.get_directory(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/filesystems.py", line 343, in get_directory
    return self.filesystem.get(from_path, local_path, recursive=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 103, in sync
    raise return_result
  File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 56, in _runner
    result[0] = await coro
                ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 614, in _get
    rpaths = await self._expand_path(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 854, in _expand_path
    out = await self._expand_path([path], recursive, maxdepth)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 877, in _expand_path
    rec = set(await self._find(p, maxdepth=maxdepth, withdirs=True))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/s3fs/core.py", line 863, in _find
    out = await self._lsdir(path, delimiter="", prefix=prefix)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/s3fs/core.py", line 744, in _lsdir
    async for c in self._iterdir(
  File "/usr/local/lib/python3.11/site-packages/s3fs/core.py", line 775, in _iterdir
    await self.set_session()
  File "/usr/local/lib/python3.11/site-packages/s3fs/core.py", line 507, in set_session
    self.key = credentials.access_key
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/credentials.py", line 409, in access_key
    self._refresh()
  File "/usr/local/lib/python3.11/site-packages/botocore/credentials.py", line 502, in _refresh
    self._protected_refresh(is_mandatory=is_mandatory_refresh)
  File "/usr/local/lib/python3.11/site-packages/botocore/credentials.py", line 518, in _protected_refresh
    metadata = self._refresh_using()
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/credentials.py", line 665, in fetch_credentials
    return self._get_cached_credentials()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/credentials.py", line 675, in _get_cached_credentials
    response = self._get_credentials()
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/credentials.py", line 909, in _get_credentials
    return client.assume_role_with_web_identity(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ClientCreatorContext' object has no attribute 'assume_role_with_web_identity'

Nate

12/04/2023, 2:21 PM

hmm, the error appears to be coming from boto, which appears to have recently released

1.33.6

- what version of boto are you using? im not sure this is directly related to prefect, but could you share the output of

prefect version

as well?

Nick Gohman

12/04/2023, 3:15 PM

We found a temporary solution. The error is in an updated version of

s3fs

(version

2023.12.0

). We pinned it down to

2023.10.0

and we were able to use our s3 blocks in prefect again in case anybody else sees this issue. I’m still digging into our use case to better understand where the best repository to raise this would be.

Nick Gohman

12/04/2023, 3:17 PM

to answer your other question, we are using the latest version of prefect (

2.14.9

)

Nate

12/04/2023, 3:17 PM

hmm - okay, thanks for sharing that!

3 Views

Open in Slack

Previous Next