https://prefect.io logo
Title
d

Dylan

02/25/2022, 10:46 PM
Hi, I’m an engineer doing a POC for Prefect. I want to get a good local setup running in docker. I tried out the docker-compose on github and am under the impression that’s used more for doing development on Prefect itself. It didn’t work and after reading back through the readme’s I tried to install the conda prefect package and run it on a simple python docker image. Now that I’ve got everything installed I’m trying to get the various components running in a single container runtime that I’ll then split out later (server, agent, ui) The idea is that this local stack be used for trying out local workflows as part of the development process, and then the docker-compose of those three ‘roles’ can be ported to helm charts and run on K8s. I’m running into a couple issues, and after sleuthing around on the GH issues I figured I’d just ask here. docker-compose
version: '3'
services:
  python-repl:
    image: prefect
    entrypoint: "/bin/bash"
    volumes:
      - "workspace:/home/workspace/"
    ports:
      - "4200:4200"
  prefect-backend-only:
    image: prefect
    command: prefect backend server && prefect agent local start && prefect server start
    volumes:
      - "workspace:/home/workspace/prefect-server"
  prefect-full-stack:
    image: prefect
    command: prefect server start
    volumes:
      - "workspace:/home/workspace/prefect-server"
volumes:
  workspace:
the dockerfile I used to build the
prefect
image above is just a python image I’ve installed conda and prefect on. I can run the
prefect backend server
command just fine but it blows up on
prefect agent local start
with
requests.exceptions.ConnectionError: HTTPConnectionPool(host='apollo', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4004761130>: Failed to establish a new connection: [Errno -2] Name or service not known'))
I could just be missing some docker-compose pieces, and ideally I could re-use the docker-compose pieces your development team uses for the local dev stack.
👋 7
k

Kevin Kho

02/25/2022, 10:58 PM
Hi Dylan, When you do
prefect agent .... start
it starts the agent that then polls the API of Prefect Server and listens to flows. You get an error because your start the agent before spinning up your server with
prefect server start
. At the very minimum, you likely need to reverse those commands. You should also be aware Prefect Cloud has 20k free task runs every month no credit card needed so you don’t have to spin up server yourself
d

Dylan

02/25/2022, 11:06 PM
thanks, what would be the path for running the server locally if we decided to go that route? It appears to look for a docker-compose when I run it inside the container runtime I installed prefect in.
root@4143c36edf59:/# prefect backend server
Backend switched to server
root@4143c36edf59:/# prefect server start
Exception caught; killing services (press ctrl-C to force)
...
...
FileNotFoundError: [Errno 2] No such file or directory: 'docker-compose'
Makes me think the
prefect backend server
is meant to be run on an OS with the
prefect
pacakge intstalled, as well as a docker agent, and that the prefect backend server is that collection of docker-compose services in the GH source?
So not within a container runtime
that’d be docker in docker, docker-ception 😄
k

Kevin Kho

02/25/2022, 11:10 PM
yes you get a box like on EC2 or GCP VM for example, install docker/docker-compose, install prefect and then do a
prefect server start --expose
and that should be it for the most part. the agent does not necessarily have to be on the same machine as the server but it can be also
prefect server start
spins up the UI, graphql api, database, etc as docker containers
I would advise against docker in docker
and thanks to sticking to the thread!
d

Dylan

02/25/2022, 11:12 PM
no problem! Yeah once I saw that error I figured something was off. Never docker in docker
So do the prefect commands roughly map to docker-compose commands then? It kinda looks that way based on the thread here and the error I saw
What’s the path for running the stack locally as a Prefect developer?
k

Kevin Kho

02/25/2022, 11:17 PM
Literally just the
prefect server start
should spin up the services
You can see this blog
prefect server start
does the docker compose for you
d

Dylan

02/25/2022, 11:21 PM
Thanks, I’ll read through the blog and look at some of the code some more. I see a bit of a chicken-egg problem if I want to run the full stack natively on our K8's stack, where I’d need a runtime with prefect installed in order to do the rest of the orchestration. I’m thinking about how to run it without the prefect command now and with native docker-compose commands, which is probably not what you all intended when designing this.
k

Kevin Kho

02/25/2022, 11:27 PM
You can use the helm chart instead of docker compose. There is a README there too
d

Dylan

02/25/2022, 11:30 PM
Awesome
thanks will give this a go
Hey Kevin, can you help clarify the verbiage around a
task
for billing purposes? Would that be any successful invocation of a python function using the
@task
decorator? Can you also provide some context around when and why someone would decide to run FOSS prefect and not use your Cloud?
k

Kevin Kho

02/28/2022, 6:32 PM
Task is anything with the task decorator or the Task library that is above 1 second. A task is tracked by our API and each task goes through at least 3 state changes
Scheduled -> Submitted -> Running -> Success
so we charge for the observability into that Task
👍 1
d

Dylan

02/28/2022, 6:32 PM
And do you have some examples of how Prefect Cloud would be used with a Dask cluster? How is the compute priced for that?
k

Kevin Kho

02/28/2022, 6:32 PM
A map over 10 items is 10 tasks because each one of those gets observability
I have examples for how to use with a Dask cluster but you provide your own and we don’t host any compute which is why our pricing is on a per task basis
You also provide the hardware that runs your own code and we don’t see your code or your data and it lives in your infrastructure
Why someone would run FOSS instead of Cloud, it mostly has to do with company security rules but to be honest we’re already about to get the SOC 2 cert and we don’t see any of your code or data by default so even using Cloud gives you a lot of privacy and security.
You get 20k task runs for free with Prefect Cloud which is way more than enough to get started with. Cloud has added features too and is more scalable than Server, which would require maintenance for production. I’d be happy to connect you to sales if you’re interested in having a chat with them
d

Dylan

02/28/2022, 6:40 PM
I’d like to understand the Prefect Cloud implications for self-hosting our code and our own compute a bit more. Do you have any architecture docs detailing that specifically? I understand patching is a trade-off with FOSS for sure and the SOC 2 news is great, so I’d like to weigh the cost of that against the benefits of Cloud.
Would that basically be single node that’d just have our user-defined python code deployed on it with some bootstrap mechanism to load the tasks to the Prefect Cloud?
k

Kevin Kho

02/28/2022, 6:43 PM
We call it the hybrid model so you will see the terminology around the website and docs.
You don’t have to load anything to Prefect Cloud. You store the code somewhere you have access to, the agent pulls it from there and then executes it
d

Dylan

02/28/2022, 6:44 PM
Thanks, I’m being a bit of a pedant right now. Is Prefect’s Cloud solution a ‘hybrid model’ or is that one of the ways in which Prefect Cloud can be used?
Ok so we’d need to launch an agent that’s basically configured to point to Prefect Cloud endpoints and our source code? Makes sense.
k

Kevin Kho

02/28/2022, 6:46 PM
The solution is just the hybrid model and we can’t host anything for now. Yes you launch an agent process, the agent polls Cloud every 10 seconds, if it finds something to run, it grabs the Flow of storage and then executes it and then reports the state back to Cloud so it’s all outbound requests
d

Dylan

02/28/2022, 6:48 PM
Excellent, and I’m assuming I can bootstrap the agent config by taking the
prefect
cli and generating the docker-compose yml, or are there docker-compose & helmchart templates for the agent stand-alone from the server stack compose generator?
I will probably take you up on connecting with the sales rep. I am going to relay info to the rest of my team and get back to you soon. Thanks for the info!
k

Kevin Kho

02/28/2022, 6:54 PM
Agent tends to be standalone from server stack. There are instructions for the Kubernetes agent to get it in cluster. And yes you just point it to your server instead of the default Cloud