Luis Muniz
07/02/2020, 2:13 PMregister()
to push a flow to a ECR (amazon) private registry, but we can't figure out how prefect picks up the credentials to do it. We know that with the command line docker build
and docker push
, you need to execute a special docker login
with aws ecr get-login...
command, but don't know how to pass this on to prefect for register(). We feel like we miss a step.from prefect.environments.storage import Docker
from strdata.strdata_poc import strdata
strdata.storage = Docker(registry_url="our_empty_aws_ecr_repository",
dockerfile="our_custom_directory_path")
strdata.register(project_name="test")
Dylan
07/02/2020, 2:24 PMflow.register
GOOGLE_APPLICATION_CREDENTIALS
to my python session when registering flows with Docker storage + Google Container Registryitay livni
07/02/2020, 2:29 PMDylan
07/02/2020, 2:29 PMLuis Muniz
07/02/2020, 2:37 PMitay livni
07/02/2020, 2:43 PMstorage = Docker(
registry_url= ecr_url # "<http://your_account.dkr.ecr.mars-east-75.amazonaws.com/get-tsx-moc-ecr|your_account.dkr.ecr.mars-east-75.amazonaws.com/get-tsx-moc-ecr>",
python_dependencies=["pandas", "sqlalchemy", "psycopg2", "boto3", "humps", "requests", "yfinance"],
dockerfile=docker_flpth,
image_name="etl-moc-img",
image_tag="latest"
)
I was looking for my old script could not find it... But what errors are you getting? The other key thing is ecr_repo_name = f"{ecr_url.replace('https://', '')}"
.Luis Muniz
07/02/2020, 2:44 PMrequests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: <http+docker://localhost/v1.40/build?t=http%3A%2F659990142216.dkr.ecr.eu-west-1.amazonaws.com%2Fstrdata-flow%2Fstrdata-poc%3A2020-07-02t14-43-38-253660-00-00&q=False&nocache=False&rm=False&forcerm=True&pull=False&dockerfile=.%2Ftmpj_kuw1de%2FDockerfile>
16:44
docker.errors.APIError: 500 Server Error: Internal Server Error ("invalid reference format")
Dylan
07/02/2020, 2:47 PMLuis Muniz
07/02/2020, 2:49 PMdocker pull
to make sure that our credentials workDylan
07/02/2020, 2:49 PMLuis Muniz
07/02/2020, 2:51 PMDylan
07/02/2020, 2:53 PMLuis Muniz
07/02/2020, 2:53 PMDylan
07/02/2020, 2:54 PMitay livni
07/02/2020, 2:54 PMDylan
07/02/2020, 2:55 PMbruno.corucho
07/02/2020, 2:57 PMDylan
07/02/2020, 2:57 PMbruno.corucho
07/02/2020, 3:06 PMLaura Lorenz (she/her)
07/02/2020, 3:12 PMitay livni
07/02/2020, 3:22 PMView push commands
?bruno.corucho
07/03/2020, 8:23 AM