Greg Roche
01/06/2021, 9:52 AMUnexpected error: KeyError('endpoint_resolver')
We have a daily flow which includes around 25 mapped S3Upload tasks, and approximately once every five flow runs, one single mapped S3Upload task fails with this error. We're not doing anything particularly complicated or novel with the task, and the flow always succeeds when restarted, so I'm curious to know if anyone else has experienced this error and knows of a fix. More details in the thread, and thanks in advance for any help πUnexpected error: KeyError('endpoint_resolver')
Traceback (most recent call last):
File "c:\my-project\.venv\lib\site-packages\prefect\engine\runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "c:\my-project\.venv\lib\site-packages\prefect\engine\task_runner.py", line 856, in get_task_run_state
value = prefect.utilities.executors.run_task_with_timeout(
File "c:\my-project\.venv\lib\site-packages\prefect\utilities\executors.py", line 298, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "c:\my-project\.venv\lib\site-packages\prefect\utilities\tasks.py", line 449, in method
return run_method(self, *args, **kwargs)
File "c:\my-project\.venv\lib\site-packages\prefect\tasks\aws\s3.py", line 147, in run
s3_client = get_boto_client("s3", credentials=credentials, **self.boto_kwargs)
File "c:\my-project\.venv\lib\site-packages\prefect\utilities\aws.py", line 55, in get_boto_client
return boto3.client(
File "c:\my-project\.venv\lib\site-packages\boto3\__init__.py", line 91, in client
return _get_default_session().client(*args, **kwargs)
File "c:\my-project\.venv\lib\site-packages\boto3\session.py", line 258, in client
return self._session.create_client(
File "c:\my-project\.venv\lib\site-packages\botocore\session.py", line 824, in create_client
endpoint_resolver = self._get_internal_component('endpoint_resolver')
File "c:\my-project\.venv\lib\site-packages\botocore\session.py", line 697, in _get_internal_component
return self._internal_components.get_component(name)
File "c:\my-project\.venv\lib\site-packages\botocore\session.py", line 923, in get_component
del self._deferred[name]
KeyError: 'endpoint_resolver'
In the flow code we're using the S3Upload task like this: (truncated code)
from prefect.tasks.aws.s3 import S3Upload
upload_to_s3 = S3Upload(bucket=Config.S3_BUCKET)
with Flow() as flow:
# ...
s3_objects = upload_to_s3.map(
data=csv_bytes,
key=s3_keys,
compression=unmapped("gzip"))
The following lines are in `config.toml`:
[cloud]
use_local_secrets = true
[AWS_CREDENTIALS]
ACCESS_KEY = "foo"
SECRET_ACCESS_KEY = "bar"
josh
01/06/2021, 2:02 PMGreg Roche
01/06/2021, 2:17 PMjosh
01/06/2021, 2:17 PMGreg Roche
01/06/2021, 4:01 PMPedro Machado
01/06/2021, 5:28 PMLocalExecutor
?Greg Roche
01/07/2021, 8:08 AMLocalDaskExecutor
for this flow (without it the flow would take an unacceptably long time to run for our use case) but I suspect that changing this to a LocalExecutor
would probably prevent this issue from happening.Pedro Machado
01/07/2021, 1:36 PM