itay livni
06/24/2020, 1:28 AMprefect.environments.storage.docker
as the mechanism to build and push containers while using Non-Docker Storage for Containerized Environments? (I ran into a bug deploying this, and wondering if this might be a reason)
flow.storage = S3(
bucket="s3-prefect-flow-storage",
secrets=["AWS_CREDENTIALS"],
)
docker = Docker(
registry_url=ecr_repo_url,
python_dependencies=[
"pandas",...],
dockerfile=docker_flpth,
image_name="annoying_docker",
image_tag="latest",
local_image=True
)
docker.build(push=True)
josh
06/24/2020, 1:44 PMenvironment.metadata
should be built independently using something like Docker’s CLI. The Docker
storage has a lot of logic baked into it around flow storage and therefore does some extra things for storing the flow that you will not need in your metadata imageitay livni
06/24/2020, 2:31 PMjosh
06/24/2020, 2:32 PMitay livni
06/24/2020, 6:22 PMjosh
06/24/2020, 6:25 PMs3-flow-storage
itay livni
06/24/2020, 7:35 PMFailed to load and execute Flow's environment: ValueError('Flow is not contained in this Storage')
I changed storage and agent configuration to labels=s3-flow-storage
.josh
06/24/2020, 7:52 PMitay livni
06/24/2020, 8:13 PMpushlog = etl_moc_flow.register(
project_name="market-on-close",
build=True,
set_schedule_active=False
)
josh
06/24/2020, 8:20 PMitay livni
06/24/2020, 8:30 PMetl_moc_flow.environment = LocalEnvironment(
metadata={"image": image}
)
Is there anything else I have to add?josh
06/24/2020, 8:31 PMitay livni
06/24/2020, 8:41 PM