Hi, I'm trying to migrate from 1.x to 2.x with Docker storage and infrastructure. The following im...

flavienbwk

09/25/2022, 6:40 PM

Hi, I'm trying to migrate from 1.x to 2.x with Docker storage and infrastructure. The following image (from blogpost) states that Prefect can store flows data in Docker images, but the documentation doesn't refer to it. Is it still possible in 2.x ? Is there any example I can find ? Thanks.

✅ 1

👍 1

Anna Geller

09/25/2022, 8:26 PM

those two might help: • https://github.com/anna-geller/prefect-docker-deployment • https://github.com/rpeden/prefect-docker-compose

flavienbwk

09/25/2022, 9:00 PM

Thank you, your repo is very useful but I don't find any reference to

Deployment.build_from_flow()

. What build_from_flow.storage value should I define for a Docker storage ? (not only infrastructure) I still get the

Flow could not be retrieved from deployment. {...} No such file or directory: '/opt/prefect/flows'

error from my agent if I don't specify a storage value. Related PR comment : https://github.com/PrefectHQ/prefect/pull/6574/files#r979469078

Anna Geller

09/25/2022, 11:13 PM

could you perhaps leverage the CLI instead, as shown in the repo? I personally think that CLI is more user-friendly and integrates better with CI/CD, but up to you

Anna Geller

09/25/2022, 11:14 PM

I also replied there

flavienbwk

09/26/2022, 8:50 PM

Thank you but I still can't make it work if I don't bind the /opt/prefect/flows volume of the agent to the one of the script registering it. If I bind it, it runs. The CLI is not a solution for my usage of Prefect in APIs. This directory exists in the agent container. Here is the error I still getting if I don't bind the Prefect directory :

flavienbwk

09/26/2022, 9:00 PM

I think I start to understand the principle of "Docker storage" in Prefect 2. In Prefect 1, stored Docker image was built directly BY Prefect with the flow placed inside. Now it is the charge of the user to place its flow files inside an image (under

/opt/prefect/flows/

) by himself, push it to registry and give it to the Prefect 2 builder. The

image

is no more a

base_image

but the image including the flow.

Anna Geller

09/26/2022, 9:38 PM

Can you share your Dockerfile

flavienbwk

09/26/2022, 10:25 PM

I've managed to make it work. What I was not understanding is the new Prefect mechanism (without the CLI, from Python code) that doesn't build the image anymore. I think it's a good move and that image build should indeed be in the hands of the user. That makes "Docker storage" more coherent. So basically in my exemple, I manually reproduce the build Prefect 1.x was automatically making before.

👍 1

Anna Geller

09/27/2022, 1:17 AM

thanks for sharing, this explains a lot! and fully agree that this is much better now in v2 🙌

3 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.