https://prefect.io logo
Title
j

Jesper Bruun Hansen

11/08/2022, 8:56 PM
Hi #prefect-docker - Does anybody have a reference to a Dockerfile with
prefecthq/prefect:2-latest
as base image, that they can run locally? Iโ€™m having hard time defining the
ENTRYPOINT
โœ… 1
r

Ryan Peden

11/08/2022, 8:59 PM
I don't have one on GitHub, but I could get one up pretty quickly. How do you want to use it locally? Knowing that will help ensure the entrypoint does what you need.
:gratitude-thank-you: 1
j

Jesper Bruun Hansen

11/08/2022, 9:00 PM
Hi Ryan, and thanks for a swift reply. Iโ€™m basically trying to build my own Docker Image (with my own pip requirements etc) and I was jus wondering if it possible to actually run that docker container locally, to test that it works ๐Ÿ™‚
So far I got
FROM prefecthq/prefect:2-latest

COPY pip/requirements.txt pip/requirements.txt
RUN pip install -r pip/requirements.txt

COPY python/ python/
WORKDIR /python/src/main

ENTRYPOINT ["python", "listings.get_listings:run_pipeline()"]
But itโ€™s not really doing it for me
r

Ryan Peden

11/08/2022, 9:06 PM
If you just want to test that it works, it might be easier to shell into it by running something like
docker run -it my_image:my_tag bash
. Then from the shell I believe you could run
python -c 'from listings.get_listings import run_pipeline; run_pipeline()'
Though if you do want to run it via the entrypoint, I think the python -c command would work there too, like:
ENTRYPOINT ["python", "-c", "'from listings.get_listings import run_pipeline; run_pipeline()'"
I haven't tested that yet, though. Going to give it a quick try now.
j

Jesper Bruun Hansen

11/08/2022, 9:15 PM
This works!
FROM prefecthq/prefect:2-latest

WORKDIR /home/root/

COPY pip/requirements.txt pip/requirements.txt
RUN pip install -r pip/requirements.txt

COPY python/ python/
WORKDIR /home/root/python/src/main

ENTRYPOINT ["python", "-m", "listings.get_listings"]
r

Ryan Peden

11/08/2022, 9:19 PM
That'll do it too, if your module runs your function instead of just defining it. Sounds like that's the case here ๐Ÿ˜„ Glad to hear that worked for you!
j

Jesper Bruun Hansen

11/08/2022, 9:19 PM
Great! So bonus question: What do I need to do now, in order to get my deployment to reference my own docker image?
I guess my goal ultimately is to have the Agent start the docker container, whenever I trigger run from Cloud - does that make sense?
r

Ryan Peden

11/08/2022, 9:27 PM
It does! There are a couple of ways. The first is running the deployment CLI with the docker-container infrastructure type as shown here, and then editing the image name in the deployment YAML before deploying it. You'll also need to set the deployment path as well because if you are including your code in the docker image, Prefect will look for it under
/opt/prefect/flows
unless you tell it otherwise.
Or, you can create a Docker Container infrastructure block using the UI or Python that specifies your custom image, and then use that pre-created block in your deployments
One note though - if you're going to use the image in a deployment, I believe you'll need to create it without setting that entrypoint. Because our Docker container infrastructure code will set your flow run ID as an environment variable in the container, and by default your deployment will run
python -m prefect.engine
which then pulls the flow information from the API based on the flow run ID, and runs the flow.
You'll also need to make sure your Docker image is present on the machine where your agent is running, or available via a Docker image registry (either public or private)
j

Jesper Bruun Hansen

11/08/2022, 9:37 PM
Hm, okay - I think it makes some sense ๐Ÿ™‚ But perhaps Iโ€™m on the wrong path here? If I want to have my flow running in a docker container, my idea was to build the docker image, push it to ECR, pull it on the Agent machine and then somehow have Cloud start that docker container for me on the remote machine (Such as a small EC2 instance or something like that).
r

Ryan Peden

11/08/2022, 9:41 PM
Got it - and I think you are on the right path! There's just one change I think you'll need to make from what I described if you want the container to start on a remote machine.
j

Jesper Bruun Hansen

11/08/2022, 9:43 PM
OK, thatโ€™s good news at least ๐Ÿ™‚ Please help me understand what that change is? Do I need to specify something else for the ENTRYPOINT or completely leave it out?
r

Ryan Peden

11/08/2022, 9:51 PM
We have another infrastructure block called ECSTask. It will run your Docker image using ECS for each flow it needs to run, backed by either EC2 or Fargate (whichever you prefer). Based on what you've mentioned here it sounds like this will accomplish what you need? Since you are using a Prefect base image, you should be able to leave the entrypoiny out and just inherit the base image entrypoint.
My colleague Anna wrote a good blog post about using ECS as part of a larger Prefect project, including deploying via CI/CD. Even if that is more than you are looking to do right now, you might find it useful ๐Ÿ˜„
j

Jesper Bruun Hansen

11/08/2022, 10:05 PM
Jeesh, that is a lot to take in - looks really cool though ๐Ÿ™‚ I do think the ECSTask is perhaps more what I need. Just a quick question: Would it be possible to pay either Anna or one of your colleagues to join us for a couple of hours, and help/teach us to get the basic level of understanding and infrastructure up and running? My guess is, that when youโ€™ve done this right the first time, building next many pipelines will be a lot easier ๐Ÿ™‚
r

Ryan Peden

11/08/2022, 10:14 PM
It's definitely easier after you've set it all up it the first time! As to your question, I need to confer with a few of my colleagues to make sure I get you in touch with the right person to provide more information ๐Ÿ™‚
Hi Jesper - As a follow up, I suggest getting in touch with our Professional Services team. You can find a couple of ways to get contact them here.
a

Anna Geller

11/08/2022, 11:56 PM
just to add, no need to override the entrypoint, the default one from the Prefect image will work, other than that, nice work so far! as Ryan mentioned, we do have paid support helping you get onboarded cs@prefect.io and I can ask any ECS specific questions too
๐Ÿ‘ 1
j

Jesper Bruun Hansen

11/09/2022, 7:21 AM
Hi @Ryan Peden and @Anna Geller - Thanks for getting back! OK Iโ€™ll reach out to your professional team. And thanks Anna for the input regarding the entrypoint ๐Ÿ‘Œ
๐Ÿ™Œ 1
โœ… 1