Good morning all Has anybody had any experience with solving Prefect Community #ask-community

Good morning all! Has anybody had any experience w...

Greg Roche

01/06/2021, 9:52 AM

Good morning all! Has anybody had any experience with solving this very transient error with the S3Upload task?

Unexpected error: KeyError('endpoint_resolver')

We have a daily flow which includes around 25 mapped S3Upload tasks, and approximately once every five flow runs, one single mapped S3Upload task fails with this error. We're not doing anything particularly complicated or novel with the task, and the flow always succeeds when restarted, so I'm curious to know if anyone else has experienced this error and knows of a fix. More details in the thread, and thanks in advance for any help 🙂

Greg Roche

01/06/2021, 9:54 AM

Full stacktrace:

Copy code

Unexpected error: KeyError('endpoint_resolver')
Traceback (most recent call last):
  File "c:\my-project\.venv\lib\site-packages\prefect\engine\runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "c:\my-project\.venv\lib\site-packages\prefect\engine\task_runner.py", line 856, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "c:\my-project\.venv\lib\site-packages\prefect\utilities\executors.py", line 298, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "c:\my-project\.venv\lib\site-packages\prefect\utilities\tasks.py", line 449, in method
    return run_method(self, *args, **kwargs)
  File "c:\my-project\.venv\lib\site-packages\prefect\tasks\aws\s3.py", line 147, in run
    s3_client = get_boto_client("s3", credentials=credentials, **self.boto_kwargs)
  File "c:\my-project\.venv\lib\site-packages\prefect\utilities\aws.py", line 55, in get_boto_client
    return boto3.client(
  File "c:\my-project\.venv\lib\site-packages\boto3\__init__.py", line 91, in client
    return _get_default_session().client(*args, **kwargs)
  File "c:\my-project\.venv\lib\site-packages\boto3\session.py", line 258, in client
    return self._session.create_client(
  File "c:\my-project\.venv\lib\site-packages\botocore\session.py", line 824, in create_client
    endpoint_resolver = self._get_internal_component('endpoint_resolver')
  File "c:\my-project\.venv\lib\site-packages\botocore\session.py", line 697, in _get_internal_component
    return self._internal_components.get_component(name)
  File "c:\my-project\.venv\lib\site-packages\botocore\session.py", line 923, in get_component
    del self._deferred[name]
KeyError: 'endpoint_resolver'

In the flow code we're using the S3Upload task like this: (truncated code)

Copy code

from prefect.tasks.aws.s3 import S3Upload

upload_to_s3 = S3Upload(bucket=Config.S3_BUCKET)

with Flow() as flow:
    # ...

    s3_objects = upload_to_s3.map(
        data=csv_bytes,
        key=s3_keys,
        compression=unmapped("gzip"))

The following lines are in `config.toml`:

Copy code

[cloud]
use_local_secrets = true

[AWS_CREDENTIALS]
ACCESS_KEY = "foo"
SECRET_ACCESS_KEY = "bar"

josh

01/06/2021, 2:02 PM

Hmm 🤔 I haven’t seen this before! Would you mind opening an issue on the repo with this information / a reproducible example?

Greg Roche

01/06/2021, 2:17 PM

Hi Josh - unfortunately I don't have a reproducible example, as I've only experienced the error occasionally (it happens less than 1% of the times that this task runs for this flow), and the error doesn't recur when the flow is re-run. If you'd still like me to open an issue for this despite the lack of reproducibility then I'm happy to 🙂

josh

01/06/2021, 2:17 PM

Yeah that would be great, a more visible record where I can reference to look into the issue would be helpful 🙂

👍 1

Greg Roche

01/06/2021, 4:01 PM

Thanks for replying, I opened an issue here: https://github.com/PrefectHQ/prefect/issues/3925

👍 1

Pedro Machado

01/06/2021, 5:28 PM

Hey Greg. I vaguely remember seeing that error when I was sharing using the s3 client in multiple threads. What executor are you using? Does it still happen if you use the

LocalExecutor

Greg Roche

01/07/2021, 8:08 AM

Hi Pedro, it certainly does seem to have something to do with the boto3 client not being thread-safe. We are using a

LocalDaskExecutor

for this flow (without it the flow would take an unacceptably long time to run for our use case) but I suspect that changing this to a

LocalExecutor

would probably prevent this issue from happening.

Pedro Machado

01/07/2021, 1:36 PM

If i remember correctly, I ended up creating the s3 client inside the map task but there may be a better approach.

132 Views

Open in Slack

Previous Next