Hi,
I am trying to write a flow that uses boto3 for interacting with AWS resources, i am installing prefect with aws extra in the docker image,
but the flow fails in tasks that try to create aws clients with:
“`botocore.exceptions.DataNotFoundError: Unable to load data for: endpoints`”
any suggestions ?
k
Kevin Kho
08/11/2021, 10:28 PM
Hey @Nadav, how do you install it with the aws extra?
pip install
or
conda
? And then are you using a DaskExecutor?
n
Nadav
08/11/2021, 10:55 PM
@Kevin Kho i install with pip and i do not use dask i use the default, i guess that is local right?
k
Kevin Kho
08/11/2021, 10:58 PM
Yeah that would be local. I haven’t seen this before. What boto3 version do you get?
Kevin Kho
08/11/2021, 11:01 PM
Maybe you can also try explicitly installing boto3 (I don’t think it’s the issue. I am more assuming that an upgrade is needed)
n
Nadav
08/11/2021, 11:10 PM
@Kevin Kho the boto3 version is .18.19,
i tried installing boto3 using the requirements file but i get the same error
Nadav
08/11/2021, 11:18 PM
when i run the docker container with -it and i run python importing boto3 and creating a client is succeeded
k
Kevin Kho
08/12/2021, 12:01 AM
Are you hosting that container on AWS then using it in your flow?
n
Nadav
08/12/2021, 5:31 AM
Yes the flow storage is docker hosted on ecr
k
Kevin Kho
08/13/2021, 1:02 AM
Oh sorry I didn’t get back to you. But I would just make sure you pull the latest image from ECR? Like maybe tag it? Because if it worked in the container locally, it should work on ECR as well.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.