Hi, I’m an engineer doing a POC for Prefect. I want to get a good local setup running in docker. I tried out the docker-compose on github and am under the impression that’s used more for doing development on Prefect itself. It didn’t work and after reading back through the readme’s I tried to install the conda prefect package and run it on a simple python docker image. Now that I’ve got everything installed I’m trying to get the various components running in a single container runtime that I’ll then split out later (server, agent, ui) The idea is that this local stack be used for trying out local workflows as part of the development process, and then the docker-compose of those three ‘roles’ can be ported to helm charts and run on K8s. I’m running into a couple issues, and after sleuthing around on the GH issues I figured I’d just ask here. docker-compose

version: '3'
services:
  python-repl:
    image: prefect
    entrypoint: "/bin/bash"
    volumes:
      - "workspace:/home/workspace/"
    ports:
      - "4200:4200"
  prefect-backend-only:
    image: prefect
    command: prefect backend server && prefect agent local start && prefect server start
    volumes:
      - "workspace:/home/workspace/prefect-server"
  prefect-full-stack:
    image: prefect
    command: prefect server start
    volumes:
      - "workspace:/home/workspace/prefect-server"
volumes:
  workspace:

the dockerfile I used to build the

prefect

image above is just a python image I’ve installed conda and prefect on. I can run the

prefect backend server

command just fine but it blows up on

prefect agent local start

with

requests.exceptions.ConnectionError: HTTPConnectionPool(host='apollo', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4004761130>: Failed to establish a new connection: [Errno -2] Name or service not known'))

I could just be missing some docker-compose pieces, and ideally I could re-use the docker-compose pieces your development team uses for the local dev stack.

Hi Dylan, When you do

prefect agent .... start

it starts the agent that then polls the API of Prefect Server and listens to flows. You get an error because your start the agent before spinning up your server with

prefect server start

. At the very minimum, you likely need to reverse those commands. You should also be aware Prefect Cloud has 20k free task runs every month no credit card needed so you don’t have to spin up server yourself

thanks, what would be the path for running the server locally if we decided to go that route? It appears to look for a docker-compose when I run it inside the container runtime I installed prefect in.

yes you get a box like on EC2 or GCP VM for example, install docker/docker-compose, install prefect and then do a

prefect server start --expose

and that should be it for the most part. the agent does not necessarily have to be on the same machine as the server but it can be also

no problem! Yeah once I saw that error I figured something was off. Never docker in docker

Literally just the

prefect server start

should spin up the services

Thanks, I’ll read through the blog and look at some of the code some more. I see a bit of a chicken-egg problem if I want to run the full stack natively on our K8's stack, where I’d need a runtime with prefect installed in order to do the rest of the orchestration. I’m thinking about how to run it without the prefect command now and with native docker-compose commands, which is probably not what you all intended when designing this.

You can use the helm chart instead of docker compose. There is a README there too

Awesome

Task is anything with the task decorator or the Task library that is above 1 second. A task is tracked by our API and each task goes through at least 3 state changes

Scheduled -> Submitted -> Running -> Success

so we charge for the observability into that Task

And do you have some examples of how Prefect Cloud would be used with a Dask cluster? How is the compute priced for that?

A map over 10 items is 10 tasks because each one of those gets observability

I’d like to understand the Prefect Cloud implications for self-hosting our code and our own compute a bit more. Do you have any architecture docs detailing that specifically? I understand patching is a trade-off with FOSS for sure and the SOC 2 news is great, so I’d like to weigh the cost of that against the benefits of Cloud.

We call it the hybrid model so you will see the terminology around the website and docs.

Thanks, I’m being a bit of a pedant right now. Is Prefect’s Cloud solution a ‘hybrid model’ or is that one of the ways in which Prefect Cloud can be used?

The solution is just the hybrid model and we can’t host anything for now. Yes you launch an agent process, the agent polls Cloud every 10 seconds, if it finds something to run, it grabs the Flow of storage and then executes it and then reports the state back to Cloud so it’s all outbound requests

Excellent, and I’m assuming I can bootstrap the agent config by taking the

prefect

cli and generating the docker-compose yml, or are there docker-compose & helmchart templates for the agent stand-alone from the server stack compose generator?

Agent tends to be standalone from server stack. There are instructions for the Kubernetes agent to get it in cluster. And yes you just point it to your server instead of the default Cloud