https://prefect.io logo
Join Slack
Powered by
Hello everyone, I'm trying to inject aws credentia...
# prefect-aws
r
Hello everyone, I'm trying to inject aws credentials as env variables into my ecs tasks using an ecs work pool env variables override. I'm collecting the credentials from an existing aws credentials block like so :
Copy code
{
  "AWS_ACCESS_KEY_ID": "{{ prefect.blocks.aws-credentials.<my-aws-block>.aws_access_key_id }}",
  "AWS_SECRET_ACCESS_KEY": "{{ prefect.blocks.aws-credentials.<my-aws-block>.aws_secret_access_key }}"
}
Unfortunately, my ecs tasks crashes at submission time with this error :
Copy code
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/workers/base.py", line 831, in _submit_run_and_capture_errors
    configuration = await self._get_configuration(flow_run)
  File "/usr/local/lib/python3.10/site-packages/prefect/workers/base.py", line 906, in _get_configuration
    configuration = await self.job_configuration.from_template_and_values(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/workers/base.py", line 125, in from_template_and_values
    populated_configuration = await resolve_block_document_references(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/templating.py", line 205, in resolve_block_document_references
    updated_value = await resolve_block_document_references(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/templating.py", line 205, in resolve_block_document_references
    updated_value = await resolve_block_document_references(
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/templating.py", line 228, in resolve_block_document_references
    block_type_slug, block_document_name = (
ValueError: too many values to unpack (expected 2)
I guess it has to do with the formatting, but I could not find the related documentation. Any help would be appreciated!
Well. Since I couldn't get this to work, I went the very ugly way:
Copy code
BOTO3_SESSION = (
    AwsCredentials.load('<my-credentials-block>')
    .get_boto3_session()
)

os.environ['AWS_ACCESS_KEY_ID'] = BOTO3_SESSION.get_credentials().access_key
os.environ['AWS_SECRET_ACCESS_KEY'] = BOTO3_SESSION.get_credentials().secret_key
👍 1
b
@Romain Vincent I am curious if you found another way of doing this since your post was from a couple months ago. I attempted to use the AWS Secret block but due to a bug/poor implementation it only works for secrets that are in binary format. My next attempt was going to be something similar to what you ended up doing. Thanks for the info in advance!
r
Hello @Bryan, sorry for the delay, I only come back every so often. My final solution was to ditch credentials altogether by providing my ecs tasks with proper roles on discreet ressources. For instance, allowing your task to read from a given bucket on s3, that kind of thing.
👍 1
b
Thanks for the response and information. That's helpful!
5 Views
Open in Slack
PreviousNext
Powered by