YSF
03/28/2023, 4:07 PMDEPLOYMENT_NAME = "local-docker-deployment"
#Import your dag definition
from dag import augmentation_pipeline
#Import prefect deployment spec library
from prefect.deployments import Deployment
from prefect.infrastructure.docker import DockerContainer
from prefect.filesystems import LocalFileSystem
docker_container_block = DockerContainer.load("example-prefect")
#fs = LocalFileSystem(basepath='C:\\Users\\ykadmin081021\\Documents\\Projects\\del2\\')
fs = LocalFileSystem.load("test-ysf")
deployment = Deployment.build_from_flow(
flow=augmentation_pipeline,
name=DEPLOYMENT_NAME,
work_queue_name="test",
infrastructure=docker_container_block,
storage=fs
)
if __name__ == "__main__":
deployment.apply()
It fails with this error: FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/C:\\Users\\ykadmin081021\\Documents\\Projects\\del2\\'
it seems to be pre-pending my provided path with /opt/prefect/
It creates the docker container successfully it seems, but then it can't access my local code. My understanding is for a docker block you have to provide storage (although is that true even if you're doing a COPY
of everything into the docker container anyways?)
Is this just always going to be incompatible with windows? Do I need to set it up as volume mount at runtime somehow? Does anyone have a working example of what I'm trying to do?Ryan Peden
03/28/2023, 4:53 PM/opt/prefect
.
If you build a custom image and COPY
your code into the /opt/prefect/flows
directory in the container, you won't need to do anything; Prefect automatically looks for your code there. In that case, all you'd need is:
DEPLOYMENT_NAME = "local-docker-deployment"
from dag import augmentation_pipeline
#Import prefect deployment spec library
from prefect.deployments import Deployment
from prefect.infrastructure.docker import DockerContainer
docker_container_block = DockerContainer.load("example-prefect")
deployment = Deployment.build_from_flow(
flow=augmentation_pipeline,
name=DEPLOYMENT_NAME,
work_queue_name="test",
infrastructure=docker_container_block,
)
If you COPY
your code file(s) to a different directory in the container, you can use the deployment's path
attribute to let Prefect know where to find it:
deployment = Deployment.build_from_flow(
flow=augmentation_pipeline,
name=DEPLOYMENT_NAME,
work_queue_name="test",
infrastructure=docker_container_block,
path="/opt/my-flows"
)
Finally, if you are running an agent on your Windows machine, you could add your Windows directory as a Docker volume so Prefect can find it. I believe the volume you'd to specify would be //c/Users/ykadmin081021/Documents/Projects/del2:/opt/prefect/flows
.
You can do this by editing the block in the Prefect UI (see attached screenshot), or in Python when creating a DockerContainer block:
my_docker_container = DockerContainer(
name="example-prefect-2",
volumes=["//c/Users/ykadmin081021/Documents/Projects/del2:/opt/prefect/flows"]
)
my_docker_container.save()
I hope this helps, but if you have more questions, please feel free to ask!mondras
05/12/2023, 5:41 PM