Hi, I'm trying to setup a workflow where I develop the pipeline locally by running the agent locally, and having it create the docker infrastructure directly on my machine. Once that's ironed out, then pushing to a serverless job. I'm using a windows machine, and am struggling to get the local docker execution working. This is my deployment file:

DEPLOYMENT_NAME = "local-docker-deployment"
#Import your dag definition
from dag import augmentation_pipeline
#Import prefect deployment spec library
from prefect.deployments import Deployment
from prefect.infrastructure.docker import DockerContainer
from prefect.filesystems import LocalFileSystem

docker_container_block = DockerContainer.load("example-prefect")
#fs = LocalFileSystem(basepath='C:\\Users\\ykadmin081021\\Documents\\Projects\\del2\\')
fs = LocalFileSystem.load("test-ysf")

deployment = Deployment.build_from_flow(
    flow=augmentation_pipeline,
    name=DEPLOYMENT_NAME,
    work_queue_name="test",
    infrastructure=docker_container_block,
    storage=fs
)


if __name__ == "__main__":
    deployment.apply()

It fails with this error:

FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/C:\\Users\\ykadmin081021\\Documents\\Projects\\del2\\'

it seems to be pre-pending my provided path with

/opt/prefect/

It creates the docker container successfully it seems, but then it can't access my local code. My understanding is for a docker block you have to provide storage (although is that true even if you're doing a

COPY

of everything into the docker container anyways?) Is this just always going to be incompatible with windows? Do I need to set it up as volume mount at runtime somehow? Does anyone have a working example of what I'm trying to do?

A LocalFileSystem block is local to wherever the flow is running - in this case, a Linux Docker container. S, Prefect will look for your flow code on the container's filesystem, and the Windows path wouldn't work even if it weren't prepended with

/opt/prefect

. If you build a custom image and

COPY

your code into the

/opt/prefect/flows

directory in the container, you won't need to do anything; Prefect automatically looks for your code there. In that case, all you'd need is:

DEPLOYMENT_NAME = "local-docker-deployment"
from dag import augmentation_pipeline
#Import prefect deployment spec library
from prefect.deployments import Deployment
from prefect.infrastructure.docker import DockerContainer

docker_container_block = DockerContainer.load("example-prefect")

deployment = Deployment.build_from_flow(
    flow=augmentation_pipeline,
    name=DEPLOYMENT_NAME,
    work_queue_name="test",
    infrastructure=docker_container_block,
)

If you

COPY

your code file(s) to a different directory in the container, you can use the deployment's

path

attribute to let Prefect know where to find it:

deployment = Deployment.build_from_flow(
    flow=augmentation_pipeline,
    name=DEPLOYMENT_NAME,
    work_queue_name="test",
    infrastructure=docker_container_block,
    path="/opt/my-flows"
)

Finally, if you are running an agent on your Windows machine, you could add your Windows directory as a Docker volume so Prefect can find it. I believe the volume you'd to specify would be

//c/Users/ykadmin081021/Documents/Projects/del2:/opt/prefect/flows

. You can do this by editing the block in the Prefect UI (see attached screenshot), or in Python when creating a DockerContainer block:

my_docker_container = DockerContainer(
    name="example-prefect-2",
    volumes=["//c/Users/ykadmin081021/Documents/Projects/del2:/opt/prefect/flows"]
)

my_docker_container.save()

I hope this helps, but if you have more questions, please feel free to ask!