https://prefect.io logo
Join Slack
Powered by
Good morning all! Has anybody had any experience w...
# prefect-community
g
Good morning all! Has anybody had any experience with solving this very transient error with the S3Upload task? 
Unexpected error: KeyError('endpoint_resolver')
We have a daily flow which includes around 25 mapped S3Upload tasks, and approximately once every five flow runs, one single mapped S3Upload task fails with this error. We're not doing anything particularly complicated or novel with the task, and the flow always succeeds when restarted, so I'm curious to know if anyone else has experienced this error and knows of a fix. More details in the thread, and thanks in advance for any help 🙂
Full stacktrace:
Copy code
Unexpected error: KeyError('endpoint_resolver')
Traceback (most recent call last):
  File "c:\my-project\.venv\lib\site-packages\prefect\engine\runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "c:\my-project\.venv\lib\site-packages\prefect\engine\task_runner.py", line 856, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "c:\my-project\.venv\lib\site-packages\prefect\utilities\executors.py", line 298, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "c:\my-project\.venv\lib\site-packages\prefect\utilities\tasks.py", line 449, in method
    return run_method(self, *args, **kwargs)
  File "c:\my-project\.venv\lib\site-packages\prefect\tasks\aws\s3.py", line 147, in run
    s3_client = get_boto_client("s3", credentials=credentials, **self.boto_kwargs)
  File "c:\my-project\.venv\lib\site-packages\prefect\utilities\aws.py", line 55, in get_boto_client
    return boto3.client(
  File "c:\my-project\.venv\lib\site-packages\boto3\__init__.py", line 91, in client
    return _get_default_session().client(*args, **kwargs)
  File "c:\my-project\.venv\lib\site-packages\boto3\session.py", line 258, in client
    return self._session.create_client(
  File "c:\my-project\.venv\lib\site-packages\botocore\session.py", line 824, in create_client
    endpoint_resolver = self._get_internal_component('endpoint_resolver')
  File "c:\my-project\.venv\lib\site-packages\botocore\session.py", line 697, in _get_internal_component
    return self._internal_components.get_component(name)
  File "c:\my-project\.venv\lib\site-packages\botocore\session.py", line 923, in get_component
    del self._deferred[name]
KeyError: 'endpoint_resolver'
In the flow code we're using the S3Upload task like this: (truncated code)
Copy code
from prefect.tasks.aws.s3 import S3Upload

upload_to_s3 = S3Upload(bucket=Config.S3_BUCKET)

with Flow() as flow:
    # ...

    s3_objects = upload_to_s3.map(
        data=csv_bytes,
        key=s3_keys,
        compression=unmapped("gzip"))
The following lines are in `config.toml`:
Copy code
[cloud]
use_local_secrets = true

[AWS_CREDENTIALS]
ACCESS_KEY = "foo"
SECRET_ACCESS_KEY = "bar"
j
Hmm 🤔 I haven’t seen this before! Would you mind opening an issue on the repo with this information / a reproducible example?
g
Hi Josh - unfortunately I don't have a reproducible example, as I've only experienced the error occasionally (it happens less than 1% of the times that this task runs for this flow), and the error doesn't recur when the flow is re-run. If you'd still like me to open an issue for this despite the lack of reproducibility then I'm happy to 🙂
j
Yeah that would be great, a more visible record where I can reference to look into the issue would be helpful 🙂
👍 1
g
Thanks for replying, I opened an issue here: https://github.com/PrefectHQ/prefect/issues/3925
👍 1
p
Hey Greg. I vaguely remember seeing that error when I was sharing using the s3 client in multiple threads. What executor are you using? Does it still happen if you use the 
LocalExecutor
?
g
Hi Pedro, it certainly does seem to have something to do with the boto3 client not being thread-safe. We are using a 
LocalDaskExecutor
for this flow (without it the flow would take an unacceptably long time to run for our use case) but I suspect that changing this to a 
LocalExecutor
would probably prevent this issue from happening.
p
If i remember correctly, I ended up creating the s3 client inside the map task but there may be a better approach.
106 Views
Open in Slack
PreviousNext
Powered by