Thread
#prefect-community
    Greg Roche

    Greg Roche

    1 year ago
    Good morning all! Has anybody had any experience with solving this very transient error with the S3Upload task?
    Unexpected error: KeyError('endpoint_resolver')
    We have a daily flow which includes around 25 mapped S3Upload tasks, and approximately once every five flow runs, one single mapped S3Upload task fails with this error. We're not doing anything particularly complicated or novel with the task, and the flow always succeeds when restarted, so I'm curious to know if anyone else has experienced this error and knows of a fix. More details in the thread, and thanks in advance for any help πŸ™‚
    Full stacktrace:
    Unexpected error: KeyError('endpoint_resolver')
    Traceback (most recent call last):
      File "c:\my-project\.venv\lib\site-packages\prefect\engine\runner.py", line 48, in inner
        new_state = method(self, state, *args, **kwargs)
      File "c:\my-project\.venv\lib\site-packages\prefect\engine\task_runner.py", line 856, in get_task_run_state
        value = prefect.utilities.executors.run_task_with_timeout(
      File "c:\my-project\.venv\lib\site-packages\prefect\utilities\executors.py", line 298, in run_task_with_timeout
        return task.run(*args, **kwargs)  # type: ignore
      File "c:\my-project\.venv\lib\site-packages\prefect\utilities\tasks.py", line 449, in method
        return run_method(self, *args, **kwargs)
      File "c:\my-project\.venv\lib\site-packages\prefect\tasks\aws\s3.py", line 147, in run
        s3_client = get_boto_client("s3", credentials=credentials, **self.boto_kwargs)
      File "c:\my-project\.venv\lib\site-packages\prefect\utilities\aws.py", line 55, in get_boto_client
        return boto3.client(
      File "c:\my-project\.venv\lib\site-packages\boto3\__init__.py", line 91, in client
        return _get_default_session().client(*args, **kwargs)
      File "c:\my-project\.venv\lib\site-packages\boto3\session.py", line 258, in client
        return self._session.create_client(
      File "c:\my-project\.venv\lib\site-packages\botocore\session.py", line 824, in create_client
        endpoint_resolver = self._get_internal_component('endpoint_resolver')
      File "c:\my-project\.venv\lib\site-packages\botocore\session.py", line 697, in _get_internal_component
        return self._internal_components.get_component(name)
      File "c:\my-project\.venv\lib\site-packages\botocore\session.py", line 923, in get_component
        del self._deferred[name]
    KeyError: 'endpoint_resolver'
    In the flow code we're using the S3Upload task like this: (truncated code)
    from prefect.tasks.aws.s3 import S3Upload
    
    upload_to_s3 = S3Upload(bucket=Config.S3_BUCKET)
    
    with Flow() as flow:
        # ...
    
        s3_objects = upload_to_s3.map(
            data=csv_bytes,
            key=s3_keys,
            compression=unmapped("gzip"))
    The following lines are in config.toml:
    [cloud]
    use_local_secrets = true
    
    [AWS_CREDENTIALS]
    ACCESS_KEY = "foo"
    SECRET_ACCESS_KEY = "bar"
    j

    josh

    1 year ago
    Hmm πŸ€” I haven’t seen this before! Would you mind opening an issue on the repo with this information / a reproducible example?
    Greg Roche

    Greg Roche

    1 year ago
    Hi Josh - unfortunately I don't have a reproducible example, as I've only experienced the error occasionally (it happens less than 1% of the times that this task runs for this flow), and the error doesn't recur when the flow is re-run. If you'd still like me to open an issue for this despite the lack of reproducibility then I'm happy to πŸ™‚
    j

    josh

    1 year ago
    Yeah that would be great, a more visible record where I can reference to look into the issue would be helpful πŸ™‚
    Greg Roche

    Greg Roche

    1 year ago
    Thanks for replying, I opened an issue here: https://github.com/PrefectHQ/prefect/issues/3925
    Pedro Machado

    Pedro Machado

    1 year ago
    Hey Greg. I vaguely remember seeing that error when I was sharing using the s3 client in multiple threads. What executor are you using? Does it still happen if you use the
    LocalExecutor
    ?
    Greg Roche

    Greg Roche

    1 year ago
    Hi Pedro, it certainly does seem to have something to do with the boto3 client not being thread-safe. We are using a
    LocalDaskExecutor
    for this flow (without it the flow would take an unacceptably long time to run for our use case) but I suspect that changing this to a
    LocalExecutor
    would probably prevent this issue from happening.
    Pedro Machado

    Pedro Machado

    1 year ago
    If i remember correctly, I ended up creating the s3 client inside the map task but there may be a better approach.