Tobias
11/09/2022, 10:48 AMUpload to S3 - maintenance flow
with a permission error.
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
I checked the permissions of dataflowops_ecs_task_role
and it does have full access to S3 "s3:*"
workflows/ecs_prefect_agent.yml
the role set for running the Create Blocks & Upload to S3
is set to dataflowops_ecs_execution_role
which does not have read/write rights to S3.
I tried adding full S3 access to this role, but still does not work.Anna Geller
11/09/2022, 12:40 PMTobias
11/09/2022, 1:25 PMprefect-orion/prod
which is the standard for the workflow. However, this might be a problem because it needs to be globally unique?
Will delete all resources and try again with a new bucket name and get back 👍FileNotFoundError: The specified bucket does not exist
Also, I cant see the bucket in S3?
I think the bucket is generated in the step called “Create Blocks & Upload to S3” which is executed without error. However, there is no resulting bucket in S3?Anna Geller
11/09/2022, 6:57 PMBen Muller
11/10/2022, 5:29 AMprefect deployment apply --upload
Anna Geller
11/10/2022, 7:48 PMBen Muller
11/10/2022, 8:20 PM2.6.7
.
As you know I am going through a migration at the moment from 1.0 -> 2.0 and this strangely only seems to occur when I deploy a new repo to our CI the first couple of times. I then run the CI step locally and it seems to fix it somehow.
Here are some logs if they are of any use. The CI step 100% has the correct credentials it needs etc. This definitely feels like some strange transient issue that keep happening.
could it be my aiobotocore/s3fs dependancies?
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/s3fs/core.py", line 111, in _error_wrapper
return await func(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/aiobotocore/client.py", line 358, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: No AWSAccessKey was presented.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 201, in coroutine_wrapper
return run_async_in_new_loop(async_fn, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 152, in run_async_in_new_loop
return anyio.run(partial(__fn, *args, **kwargs))
File "/usr/local/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
return future.result()
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
return await func(*args)
File "/usr/local/lib/python3.9/site-packages/prefect/cli/deployment.py", line 519, in apply
file_count = await deployment.upload_to_storage()
File "/usr/local/lib/python3.9/site-packages/prefect/deployments.py", line 576, in upload_to_storage
file_count = await self.storage.put_directory(
File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 481, in put_directory
return await self.filesystem.put_directory(
File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 358, in put_directory
self.filesystem.put_file(f, fpath, overwrite=True)
File "/usr/local/lib/python3.9/site-packages/fsspec/asyn.py", line 111, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/fsspec/asyn.py", line 96, in sync
raise return_result
File "/usr/local/lib/python3.9/site-packages/fsspec/asyn.py", line 53, in _runner
result[0] = await coro
File "/usr/local/lib/python3.9/site-packages/s3fs/core.py", line 1056, in _put_file
await self._call_s3(
File "/usr/local/lib/python3.9/site-packages/s3fs/core.py", line 338, in _call_s3
return await _error_wrapper(
File "/usr/local/lib/python3.9/site-packages/s3fs/core.py", line 138, in _error_wrapper
raise err
PermissionError: No AWSAccessKey was presented.
An exception occurred.
ERROR: 1
Failed to deploy prefect flows!
Anna Geller
11/14/2022, 1:21 AMBen Muller
11/14/2022, 1:22 AMAnna Geller
11/14/2022, 1:24 AMBen Muller
11/14/2022, 1:27 AMAnna Geller
11/14/2022, 2:40 AMBen Muller
11/14/2022, 2:41 AM--path
arg in prefect build deployment
it seems to work 🤷Anna Geller
11/14/2022, 2:43 AMBen Muller
11/14/2022, 2:50 AMAnna Geller
11/14/2022, 1:22 PMAndrew
12/16/2022, 6:32 PMBucket Not Found
error when you specify a project/file.py:function_name
rather than project:file.py:FlowName
, as well.Anna Geller
12/16/2022, 6:34 PM