https://prefect.io logo
Title
y

YSF

03/28/2023, 4:07 PM
Hi, I'm trying to setup a workflow where I develop the pipeline locally by running the agent locally, and having it create the docker infrastructure directly on my machine. Once that's ironed out, then pushing to a serverless job. I'm using a windows machine, and am struggling to get the local docker execution working. This is my deployment file:
DEPLOYMENT_NAME = "local-docker-deployment"
#Import your dag definition
from dag import augmentation_pipeline
#Import prefect deployment spec library
from prefect.deployments import Deployment
from prefect.infrastructure.docker import DockerContainer
from prefect.filesystems import LocalFileSystem

docker_container_block = DockerContainer.load("example-prefect")
#fs = LocalFileSystem(basepath='C:\\Users\\ykadmin081021\\Documents\\Projects\\del2\\')
fs = LocalFileSystem.load("test-ysf")

deployment = Deployment.build_from_flow(
    flow=augmentation_pipeline,
    name=DEPLOYMENT_NAME,
    work_queue_name="test",
    infrastructure=docker_container_block,
    storage=fs
)


if __name__ == "__main__":
    deployment.apply()
It fails with this error:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/C:\\Users\\ykadmin081021\\Documents\\Projects\\del2\\'
it seems to be pre-pending my provided path with
/opt/prefect/
It creates the docker container successfully it seems, but then it can't access my local code. My understanding is for a docker block you have to provide storage (although is that true even if you're doing a
COPY
of everything into the docker container anyways?) Is this just always going to be incompatible with windows? Do I need to set it up as volume mount at runtime somehow? Does anyone have a working example of what I'm trying to do?
@Taylor Curran, tagging
r

Ryan Peden

03/28/2023, 4:53 PM
A LocalFileSystem block is local to wherever the flow is running - in this case, a Linux Docker container. S, Prefect will look for your flow code on the container's filesystem, and the Windows path wouldn't work even if it weren't prepended with
/opt/prefect
. If you build a custom image and
COPY
your code into the
/opt/prefect/flows
directory in the container, you won't need to do anything; Prefect automatically looks for your code there. In that case, all you'd need is:
DEPLOYMENT_NAME = "local-docker-deployment"
from dag import augmentation_pipeline
#Import prefect deployment spec library
from prefect.deployments import Deployment
from prefect.infrastructure.docker import DockerContainer

docker_container_block = DockerContainer.load("example-prefect")

deployment = Deployment.build_from_flow(
    flow=augmentation_pipeline,
    name=DEPLOYMENT_NAME,
    work_queue_name="test",
    infrastructure=docker_container_block,
)
If you
COPY
your code file(s) to a different directory in the container, you can use the deployment's
path
attribute to let Prefect know where to find it:
deployment = Deployment.build_from_flow(
    flow=augmentation_pipeline,
    name=DEPLOYMENT_NAME,
    work_queue_name="test",
    infrastructure=docker_container_block,
    path="/opt/my-flows"
)
Finally, if you are running an agent on your Windows machine, you could add your Windows directory as a Docker volume so Prefect can find it. I believe the volume you'd to specify would be
//c/Users/ykadmin081021/Documents/Projects/del2:/opt/prefect/flows
. You can do this by editing the block in the Prefect UI (see attached screenshot), or in Python when creating a DockerContainer block:
my_docker_container = DockerContainer(
    name="example-prefect-2",
    volumes=["//c/Users/ykadmin081021/Documents/Projects/del2:/opt/prefect/flows"]
)

my_docker_container.save()
I hope this helps, but if you have more questions, please feel free to ask!
m

mondras

05/12/2023, 5:41 PM
@Matt Delacour