itay livni
05/16/2020, 9:19 PMS3Result
and receiving a
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist
Which upon further research - it can be anything including a permission error. (I tried different buckets with settings) The credentials are stored as AWS_CREDENTIALS in prefect cloud. With the config.toml set to use cloud secrets
[cloud]
use_local_secrets = false
Switching back to result_handler
argument with S3Result
subclass did work, . And combining result handler
with target
does not. Is there something different in the way that credentials are handled between result
and result_handler
?
The new prefect is really nice 🙂Chris White
05/16/2020, 9:24 PMresult_handler
and the result
?itay livni
05/16/2020, 9:24 PMtsx_imb_res = S3Result(bucket="tsx-moc-bcp")
@task(
max_retries=3,
retry_delay=timedelta(seconds=1), # In production this will be change to 20 minutes
result_handler=tsx_imb_res,
target="{task_name}-{today}",
state_handlers=[imb_handler, error_handler]
)
target
commented outChris White
05/16/2020, 9:26 PMresult_handler
kwarg is now deprecated, so you should instead try:
...
result=tsx_imb_res,
...
itay livni
05/16/2020, 9:26 PMChris White
05/16/2020, 9:28 PMexists
logic on the S3Result
type. Would you mind sharing this example code + the traceback you’re seeing? Sorry about that!itay livni
05/16/2020, 9:32 PMtarget
right?
[2020-05-16 21:31:37] INFO - prefect.FlowRunner | Beginning Flow run for 'Our first flow'
[2020-05-16 21:31:37] INFO - prefect.FlowRunner | Starting flow run.
[2020-05-16 21:31:37] INFO - prefect.TaskRunner | Task 'tsx_url': Starting task run...
[2020-05-16 21:31:37] INFO - prefect.TaskRunner | Task 'tsx_url': finished task run for task with final state: 'Success'
[2020-05-16 21:31:37] INFO - prefect.TaskRunner | Task 'get_tsx_moc_imb': Starting task run...
[2020-05-16 21:31:38] ERROR - prefect.TaskRunner | Unexpected error: NoSuchKey('An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.')
Traceback (most recent call last):
File "/home/ilivni/miniconda3/envs/py37moc/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/home/ilivni/miniconda3/envs/py37moc/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 651, in check_target
if result.exists(target, **prefect.context):
File "/home/ilivni/miniconda3/envs/py37moc/lib/python3.7/site-packages/prefect/engine/results/s3_result.py", line 167, in exists
Bucket=self.bucket, Key=location.format(**kwargs)
File "/home/ilivni/miniconda3/envs/py37moc/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/ilivni/miniconda3/envs/py37moc/lib/python3.7/site-packages/botocore/client.py", line 626, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
[2020-05-16 21:31:38] INFO - prefect.TaskRunner | Task 'get_tsx_moc_imb': finished task run for task with final state: 'Skipped'
[2020-05-16 21:31:38] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
target="{task_name}-{today}",
Chris White
05/16/2020, 9:35 PMtarget
specification; I think this exception catching logic here is flawed: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/engine/results/s3_result.py#L169boto3
or something, we’ll need to investigate a little deeperitay livni
05/16/2020, 9:42 PMChris White
05/16/2020, 9:42 PMMarvin
05/16/2020, 9:42 PMitay livni
05/21/2020, 2:28 AM