https://prefect.io logo
Join Slack
Powered by
Hi all, I'm trying to run a deployment in a local ...
# ask-community
j
Hi all, I'm trying to run a deployment in a local docker work pool using Prefect 3.1.2, self-hosted. My ultimate goal is to write, test and deploy scripts to run in a kubernetes cluster on EKS. But to avoid setting up a cluster on my development machine, I've set up a docker-compose environment containing everything I need to develop and run scripts. I'm able to serve() flows to Prefect in my docker containers just fine. But to simulate the scenario where my flows will be deployed to a dynamic infrastructure while, as I said, avoiding setting up a local kubernetes cluster, I'm trying to deploy() them to a local docker work pool. I'm also hoping to avoid building the image, and if possible, even starting the container on every flow run. So my deploy looks like this:
Copy code
flow.deploy(
        name="my-code",
        work_pool_name="my-docker-work-pool",
        image="my-image:latest",
        build=False,
        push=False
)
Does this even make sense as an approach? Assuming it is, the deploy succeeds, but on running the flow, I get: docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.47/images/create?tag=latest&fromImage=my-image: Not Found ("pull access denied for my-image, repository does not exist or may require 'docker login': denied: requested access to the resource is denied") I'm running the Prefect server in a local virtual python environment which should have access to my local Mac docker environment. Is there something I need to do in my docker or work pool configuration to get this to work? Or am I thinking about it all wrong? Thanks!
w
this makes sense as an approach, I see in your deploy that you do not build the image (build=False), does this mean that my-image is being built elsewhere? How is the my-image being generated? This may be why you are facing the 404 image not found
j
yes it is being built as part of my docker compose network, which is why I don't want Prefect to build it
I might have figured it out. I think I needed to be running a local image registry, which I am now doing on port 5001 (port 5000 conflicted on the Mac). To run a local registry: 
docker run -d -p 5001:5000 --name local-registry registry:2
tag my-image: 
docker tag spark-dev localhost:5001/my-image
push my-image to the local registry: 
docker push localhost:5001/my-image
My deployment:
Copy code
flow.deploy(
        name="my-code",
        work_pool_name="my-docker-work-pool",
        image="localhost:5001/my-image:latest",
        build=False,
        push=False
)
The deployment now appears to be running successfully in that the runs appear to complete, according to their logs and to my local docker, but the print statements are not logging despite 
@flow(log_prints=True)
and the runs remain in a Pending state. Progress, but not quite there. I'll try to simplify my image and go from there...
10 Views
Open in Slack
PreviousNext
Powered by