Ankit
07/09/2020, 7:52 AMprefect ui+server
as well. My question is prefect server
requires docker and docker-compose
as mentioned in the docs. So I can't run it inside the docker container for prefect, right? How would I go about doing this then? Any help would be appreciated.
TIAnicholas
07/09/2020, 8:04 AMAnkit
07/09/2020, 8:11 AMdocker-compose.yml
file from the repo. Would put all the containers in the same network, and hopefully that should work.
I am not sure if what I am doing is exactly what you suggested.nicholas
07/09/2020, 8:21 AMGreg Desmarais
07/09/2020, 10:16 PMAnkit
07/10/2020, 5:40 AMGreg Desmarais
07/10/2020, 1:43 PMnicholas
07/10/2020, 1:44 PMGreg Desmarais
07/10/2020, 1:44 PMnicholas
07/10/2020, 1:44 PMGreg Desmarais
07/10/2020, 1:46 PMclass RightsizeFlow(Flow):
def __init__(self, *args, **kwargs):
super(RightsizeFlow, self).__init__(*args, **kwargs)
############## Storage ecr docker flow ##############
# aws configuration
aws_ecr_repo_name = "prefect_flows"
aws_region = DEFAULT_REGION
# See <https://github.com/awslabs/amazon-ecr-credential-helper>
# 1. Reset Auth (hackish)
# dkr_ecr_scrt = PrefectSecret("docker_ecr_login").run()
# get_ecr_auth_token = ShellTask(helper_script="cd ~")
# ecr_auth_token = get_ecr_auth_token.run(command=dkr_ecr_scrt)
ecr_client = boto3.client('ecr', region_name=aws_region)
ecr_token = ecr_client.get_authorization_token()
# # Decode the aws token
username, password = base64.b64decode(ecr_token['authorizationData'][0]['authorizationToken'])\
.decode().split(':')
ecr_registry_url = ecr_token['authorizationData'][0]['proxyEndpoint']
# # Registry URL for prefect or docker push
flow_registry_url = os.path.join(ecr_registry_url.replace('https://', ''), aws_ecr_repo_name)
# see <https://docs.prefect.io/api/latest/environments/storage.html#docker>
image_name = re.sub(r'[^a-zA-Z0-9_.-]', '_', self.name).lower()
# docker tag prefect_flows:latest ########.<http://dkr.ecr.us-east-1.amazonaws.com/prefect_flows:latest|dkr.ecr.us-east-1.amazonaws.com/prefect_flows:latest>
# registry_url = "########.<http://dkr.ecr.us-east-1.amazonaws.com/prefect_flows|dkr.ecr.us-east-1.amazonaws.com/prefect_flows>"
# local_image (bool, optional): an optional flag whether or not to use a local docker image,
# if True then a pull will not be attempted
use_local_image = True
storage = Docker(
base_image='celsiustx/rightsize_99_standard_py37',
image_name=image_name,
local_image=use_local_image,
registry_url=flow_registry_url
)
self.storage = storage
nicholas
07/10/2020, 1:48 PMGreg Desmarais
07/10/2020, 1:59 PMnicholas
07/10/2020, 2:00 PMGreg Desmarais
07/10/2020, 2:01 PMnicholas
07/10/2020, 2:02 PMGreg Desmarais
07/10/2020, 2:02 PMaws ecr get-login --no-include-email --region us-east-1
My next step (after meeting Friday) is to integrate this call into the script - possibly as a ShellTask?