i rather not have to take the manual setp of loading aws cre

Question

i rather not have to take the manual setp of loading aws credentials into prefect can i do something like this ``` credentials AWS ACCESS KEY ID $AWS ACCESS KEY ID AWS SECRET ACCESS KEY $AWS SECRET ACCESS KEY AWS DEFAULT REGION $AWS REGION ``` im getting `TypeError client got an unexpected keyword argument AWS ACCESS KEY ID ` so hoping i just got the wrong key for this

Jake Kaplan · Answer

can you paste the full traceback

sqlboi · Answer

sure ```Successfully installed mypy boto3 s3 1 28 36 mypy boto3 secretsmanager 1 28 36 prefect aws 0 3 7 tenacity 8 2 3 Traceback most recent call last File job venv lib python3 9 site packages prefect deployments steps core py line 122 in run steps step output = await run step step upstream outputs File job venv lib python3 9 site packages prefect deployments steps core py line 93 in run step result = await from async call soon in new thread File job venv lib python3 9 site packages prefect internal concurrency calls py line 291 in aresult return await asyncio wrap future self future File job venv lib python3 9 site packages prefect internal concurrency calls py line 315 in run sync result = self fn self args self kwargs File job venv lib python3 9 site packages prefect aws deployments steps py line 99 in push to s3 client = boto3 client File job venv lib python3 9 site packages boto3 init py line 92 in client return get default session client args kwargs TypeError client got an unexpected keyword argument AWS ACCESS KEY ID The above exception was the direct cause of the following exception Traceback most recent call last File job venv lib python3 9 site packages prefect cli utilities py line 41 in wrapper return fn args kwargs File job venv lib python3 9 site packages prefect utilities asyncutils py line 255 in coroutine wrapper return call File job venv lib python3 9 site packages prefect internal concurrency calls py line 382 in call return self result File job venv lib python3 9 site packages prefect internal concurrency calls py line 282 in result return self future result timeout=timeout File job venv lib python3 9 site packages prefect internal concurrency calls py line 168 in result return self get result File usr lib python3 9 concurrent futures base py line 389 in get result raise self exception File job venv lib python3 9 site packages prefect internal concurrency calls py line 345 in run async result = await coro File job venv lib python3 9 site packages prefect cli deploy py line 249 in deploy await run single deploy File job venv lib python3 9 site packages prefect client utilities py line 51 in with injected client return await fn args kwargs File job venv lib python3 9 site packages prefect cli deploy py line 511 in run single deploy await run steps push steps step outputs print function=app console print File job venv lib python3 9 site packages prefect deployments steps core py line 150 in run steps raise StepExecutionError f Encountered error while running fqn from exc prefect deployments steps core StepExecutionError Encountered error while running prefect aws deployments steps push to s3 An exception occurred ``` another jake

Jake Kaplan · Answer

can you try `aws access key id` lowercase for the 3 inputs

Jake Kaplan · Answer

and yes there are dozens of us

sqlboi · Answer

thank you other jake i think lowercase is the ticket only thing is its not likeing aws default region is there another key for that i got that from here <https linen prefect io t 2434789 u02h1a95xdw u01qej9pp53 u01pcj2dmv0 thanks everyone for the >

Jake Kaplan · Answer

I believe the client it s using is this one <https boto3 amazonaws com v1 documentation api latest reference core session html boto3 session Session client> so it would be `region name` I think for the param name

sqlboi · Answer

awesome thanks

sqlboi · Answer

do i have to use all those parameters cuz now its telling me i need an endpoint

sqlboi · Answer

```Traceback most recent call last File job venv lib python3 9 site packages prefect deployments steps core py line 122 in run steps step output = await run step step upstream outputs File job venv lib python3 9 site packages prefect deployments steps core py line 93 in run step result = await from async call soon in new thread File job venv lib python3 9 site packages prefect internal concurrency calls py line 291 in aresult return await asyncio wrap future self future File job venv lib python3 9 site packages prefect internal concurrency calls py line 315 in run sync result = self fn self args self kwargs File job venv lib python3 9 site packages prefect aws deployments steps py line 99 in push to s3 client = boto3 client File job venv lib python3 9 site packages boto3 init py line 92 in client return get default session client args kwargs File job venv lib python3 9 site packages boto3 session py line 299 in client return self session create client File job venv lib python3 9 site packages botocore session py line 997 in create client client = client creator create client File job venv lib python3 9 site packages botocore client py line 159 in create client client args = self get client args File job venv lib python3 9 site packages botocore client py line 490 in get client args return args creator get client args File job venv lib python3 9 site packages botocore args py line 137 in get client args endpoint = endpoint creator create endpoint File job venv lib python3 9 site packages botocore endpoint py line 402 in create endpoint raise ValueError Invalid endpoint s endpoint url ValueError Invalid endpoint The above exception was the direct cause of the following exception Traceback most recent call last File job venv lib python3 9 site packages prefect cli utilities py line 41 in wrapper return fn args kwargs File job venv lib python3 9 site packages prefect utilities asyncutils py line 255 in coroutine wrapper return call File job venv lib python3 9 site packages prefect internal concurrency calls py line 382 in call return self result File job venv lib python3 9 site packages prefect internal concurrency calls py line 282 in result return self future result timeout=timeout File job venv lib python3 9 site packages prefect internal concurrency calls py line 168 in result return self get result File usr lib python3 9 concurrent futures base py line 389 in get result raise self exception File job venv lib python3 9 site packages prefect internal concurrency calls py line 345 in run async result = await coro File job venv lib python3 9 site packages prefect cli deploy py line 249 in deploy await run single deploy File job venv lib python3 9 site packages prefect client utilities py line 51 in with injected client return await fn args kwargs File job venv lib python3 9 site packages prefect cli deploy py line 511 in run single deploy await run steps push steps step outputs print function=app console print File job venv lib python3 9 site packages prefect deployments steps core py line 150 in run steps raise StepExecutionError f Encountered error while running fqn from exc prefect deployments steps core StepExecutionError Encountered error while running prefect aws deployments steps push to s3 An exception occurred ```

Jake Kaplan · Answer

looks like you re not passing a bucket name ```ValueError Invalid endpoint ```

sqlboi · Answer

using ```pull prefect aws deployments steps pull from s3 requires prefect aws gt =0 3 0 bucket mybucket name folder push code folder credentials aws access key id $AWS ACCESS KEY ID aws secret access key $AWS SECRET ACCESS KEY region name $AWS REGION ```

sqlboi · Answer

tried with s3 too

sqlboi · Answer

wait a sec am i supposed to create a folder in s3 or is it creating it for me because i figured i could have it create a folder for each flow

sqlboi · Answer

that s a no cuz i created the folder and tested

Jake Kaplan · Answer

sorry I was mistaken about the bucket this is all it is doing sending your params straight to boto from what you pass boto needs to be able to construct a url like ` region code `

Jake Kaplan · Answer

so it looks like it s getting ` ` for the region

Jake Kaplan · Answer

are you sure that env `$AWS REGION` is a valid env var

sqlboi · Answer

ok thats weird because its set AWS REGION= us east 1

sqlboi · Answer

ur definitely right tho because i just hard coded us east 1 into there and it worked

sqlboi · Answer

< Jake Kaplan> question regarding aws arn i deployed and tried to run manually wondering if i need to create a bucket without `Server side encryption with AWS Key Management Service keys SSE KMS ` because i didnt see it in the boto link u sent and got ```Flow could not be retrieved from deployment Traceback most recent call last File usr local lib python3 11 site packages prefect deployments steps core py line 122 in run steps step output = await run step step upstream outputs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages prefect deployments steps core py line 93 in run step result = await from async call soon in new thread ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages prefect internal concurrency calls py line 291 in aresult return await asyncio wrap future self future ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages prefect internal concurrency calls py line 315 in run sync result = self fn self args self kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages prefect aws deployments steps py line 187 in pull from s3 for result in paginator paginate Bucket=bucket Prefix=folder File usr local lib python3 11 site packages botocore paginate py line 269 in iter response = self make request current kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages botocore paginate py line 357 in make request return self method current kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages botocore client py line 535 in api call return self make api call operation name kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages botocore client py line 980 in make api call raise error class parsed response operation name botocore exceptions ClientError An error occurred AccessDenied when calling the ListObjectsV2 operation Access Denied The above exception was the direct cause of the following exception Traceback most recent call last File usr local lib python3 11 site packages prefect engine py line 394 in retrieve flow then begin flow run flow = await load flow from flow run flow run client=client ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages prefect client utilities py line 51 in with injected client return await fn args kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages prefect deployments deployments py line 206 in load flow from flow run output = await run steps deployment pull steps ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages prefect deployments steps core py line 150 in run steps raise StepExecutionError f Encountered error while running fqn from exc prefect deployments steps core StepExecutionError Encountered error while running prefect aws deployments steps pull from s3```

Jake Kaplan · Answer

```botocore exceptions ClientError An error occurred AccessDenied when calling the ListObjectsV2 operation Access Denied```

sqlboi · Answer

yeah i have a feeling its because the kms

sqlboi · Answer

testing out

Jake Kaplan · Answer

Maybe I m not sure how that presents itself I would make sure your credentials have permissions to read from the bucket I d look up using boto3 directly to read from a bucket to debug your permission issues Once you have that the values back to prefect are just plugged in

sqlboi · Answer

ok thanks and yeah i changed credentials and am using AdministratorAccess on a bucket without kms so something fishy

sqlboi · Answer

i have the credentials in env which did not go into the bucket so could that be the reason like all the files got loaded into s3 just when i run the flow it errors out

sqlboi · Answer

< Jake Kaplan> figured it out cant map push values from pull in yml thank you