Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

image.png

Hi,

I'm trying to migrate from 1.x to 2.x with Docker storage and infrastructure.

The following image <https://towardsdatascience.com/create-robust-data-pipelines-with-prefect-docker-and-github-12b231ca6ed2|(from blogpost)> states that Prefect can store flows data in Docker images, but the <https://docs.prefect.io/concepts/storage/#supported-storage-blocks|documentation> doesn't refer to it.

Is it still possible in 2.x ?
Is there any example I can find ?

Thanks.

those two might help:
• <https://github.com/anna-geller/prefect-docker-deployment>
• <https://github.com/rpeden/prefect-docker-compose>

Thank you, your repo is very useful but I don't find any reference to `Deployment.build_from_flow()` . What build_from_flow.storage value should I define for a Docker *storage* ? (not only infrastructure)

I still get the `Flow could not be retrieved from deployment. {...} No such file or directory: '/opt/prefect/flows'`  error from my agent if I don't specify a *storage* value.

Related PR comment : <https://github.com/PrefectHQ/prefect/pull/6574/files#r979469078>

could you perhaps leverage the CLI instead, as shown in the repo? I personally think that CLI is more user-friendly and integrates better with CI/CD, but up to you

Thank you but I still can't make it work if I don't bind the /opt/prefect/flows volume of the agent to the one of the script registering it. If I bind it, it runs.

The CLI is not a solution for my usage of Prefect in APIs.

This directory exists in the agent container.

Here is the error I still getting if I don't bind the Prefect directory :

I think I start to understand the principle of "Docker storage" in Prefect 2. In Prefect 1, stored Docker image was built directly BY  Prefect with the flow placed inside. Now it is the charge of the user to place its flow files inside an image (under `/opt/prefect/flows/`) by himself, push it to registry and give it to the Prefect 2 builder. The `image` is no more a `base_image` but the image including the flow.

I've managed to <https://github.com/flavienbwk/prefect-docker-compose/blob/98d110d2994fd17dcf7e107eab1aa5cc94b3dd02/client_docker/app/weather.py#L59|make it work>.
What I was not understanding is the new Prefect mechanism (without the CLI, from Python code) that doesn't build the image anymore. I think it's a good move and that image build should indeed be in the hands of the user. That makes "Docker storage" more coherent.

So basically in my exemple, I manually reproduce the build <https://github.com/flavienbwk/prefect-docker-compose/blob/e758a498d5819550a9b926b0bf9bb4e9c85574d1/client_docker/weather.py#L42|Prefect 1.x was automatically making> before.

thanks for sharing, this explains a lot! and fully agree that this is much better now in v2 :raised_hands: