https://prefect.io logo
Title
i

itay livni

06/24/2020, 1:28 AM
Hi - I am being lazy about re-configuring another docker build push interface. Is it possible to use
prefect.environments.storage.docker
as the mechanism to build and push containers while using Non-Docker Storage for Containerized Environments? (I ran into a bug deploying this, and wondering if this might be a reason)
flow.storage = S3(
    bucket="s3-prefect-flow-storage",
    secrets=["AWS_CREDENTIALS"],
    )
docker = Docker(
    registry_url=ecr_repo_url,
    python_dependencies=[
        "pandas",...],
    dockerfile=docker_flpth,
    image_name="annoying_docker",
    image_tag="latest",
    local_image=True
    )
docker.build(push=True)
j

josh

06/24/2020, 1:44 PM
Hi @itay livni personally I think the image you set in the
environment.metadata
should be built independently using something like Docker’s CLI. The
Docker
storage has a lot of logic baked into it around flow storage and therefore does some extra things for storing the flow that you will not need in your metadata image
👍 1
:upvote: 1
i

itay livni

06/24/2020, 2:31 PM
Although I do like the health checks. It's like a pre-push hook.
j

josh

06/24/2020, 2:32 PM
Yeah they are nice 🙂 you could call them directly https://github.com/PrefectHQ/prefect/blob/master/src/prefect/environments/storage/_healthcheck.py although some might not be viable without being inside your flow’s image
i

itay livni

06/24/2020, 6:22 PM
@josh 1. I built and pushed the docker image to ecr using the aws cli (docker-gist) 2. Ran the deploy script (gist) 3. Started the `FargateAgent (start-agent-gist) successfully. 4. Invoked a quick run The flow got scheduled but did not run. No ECS tasks began and the agent doesn't seem to know that a quick run was invoked. -- Thanks
j

josh

06/24/2020, 6:25 PM
The default label is
s3-flow-storage
i

itay livni

06/24/2020, 7:35 PM
@josh Thanks. That worked. The new error from the Cloud is
Failed to load and execute Flow's environment: ValueError('Flow is not contained in this Storage')
I changed storage and agent configuration to
labels=s3-flow-storage
.
j

josh

06/24/2020, 7:52 PM
How are you registering the flow?
i

itay livni

06/24/2020, 8:13 PM
@josh
pushlog = etl_moc_flow.register(
    project_name="market-on-close", 
    build=True,
    set_schedule_active=False
    )
j

josh

06/24/2020, 8:20 PM
Hmm interesting, and this flow has both s3 storage and an image name in the environment metadata? FWIW I just did this and it was working for me 🙂
i

itay livni

06/24/2020, 8:30 PM
@josh the enviroment is simply configured like this;
etl_moc_flow.environment = LocalEnvironment(
    metadata={"image": image}
    )
Is there anything else I have to add?
j

josh

06/24/2020, 8:31 PM
No that should be it! Would you mind opening an issue so we can track?
i

itay livni

06/24/2020, 8:41 PM
Will do. Thanks.