Hey everyone! I wanted to try out the new feature: Non-blocking submission of flow runs to the

Runner

webserver released in version 2.14.17, but wasn't able to get the runner setup working. In the community, I could see others already using it and hence, wanted to know what I was doing wrong.

@Prathamesh Chavan Can you post your docker compose config or similar? I think we'd need some more detail to help than what you've described. I have the web runner running with a docker compose setup and 50 workers running concurrenttly at the same time (though keep having the server thread crash after a bunch of successful flow runs about 26-28 for reasons I'm still trying to puzzle out in my particular use case).

Hi @Daryl I am hosting the prefect server as a process and starting it using the command:

prefect server start

Mostly all of the settings are default, apart from setting

PREFECT_EXPERIMENTAL_ENABLE_EXTRA_RUNNER_ENDPOINTS

and

PREFECT_RUNNER_SERVER_ENABLE

to true

@Prathamesh Chavan I'd strongly recommend dockerizing this rather than trying to run on a local machine or server as a pip install. Here's my

docker-compose

which uses the standard prefect docker images. Note it's got extra pips installed for my particular use case in astronomy and w sqlalchmey. If you don't want to bother with the postgres install commnet out the

db

stanza and comment out the postgres

PREFECT_API_DATABASE_CONNECTION_URL

and uncommnet the one below it for sqlite. Note that you will need a

.env

file in the project root if you use the poastgres to set the variables for user, pass, and db. Iif you're just getting started it's probablyt better to use the sqlite to simlpify things.). You'll also prolly have to comment out the

/data/staging

volumes as well unless you want to do local file operations etc.

version: "3.9"
services:
  server:
    image: prefecthq/prefect:2.16.3-python3.11
    restart: always
    volumes:
      - prefect:/root/.prefect
      - /data/staging:/data/staging
    entrypoint: ["/bin/bash", "-c"]
    environment:
      # - PREFECT_SERVER_API_HOST=0.0.0.0
      - PREFECT_SERVER_API_HOST=127.0.0.1
      - PREFECT_API_DATABASE_CONNECTION_URL=postgresql+asyncpg://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:5432/${POSTGRES_DB}
      # - PREFECT_API_DATABASE_CONNECTION_URL=sqlite+aiosqlite:////root/.prefect/prefect.db
      - EXTRA_PIP_PACKAGES=astropy prefect-sqlalchemy psycopg2-binary datetime httpx
    command: ["prefect server start --host 0.0.0.0"]
    ports:
      - "4200:4200"

  worker:
    image: prefecthq/prefect:2.16.3-python3.11
    restart: always
    volumes:
      - .:/app
      - /data/staging:/data/staging
    working_dir: /app
    command: ["sh", "-c", "sleep 3 && python pipeline.py"]
    environment:
      - PREFECT_API_URL=<http://server:4200/api>
      - PREFECT_EXPERIMENTAL_ENABLE_EXTRA_RUNNER_ENDPOINTS=True
      - PREFECT_RUNNER_SERVER_HOST=127.0.0.1
      - EXTRA_PIP_PACKAGES=astropy prefect-sqlalchemy psycopg2-binary datetime httpx
    depends_on:
      - server
    ports:
      - "8080:8080"

  db:
    image: postgres:15.4-alpine
    container_name: prefect-db
    restart: always
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
    volumes:
      - db:/var/lib/postgresql/data
      # - /container-volumes/pg-volume:/var/lib/postgresql/data
    ports:
      - "5432:5432"

volumes:
  prefect:
  db:
networks:
  prefect-network:
    driver: bridge

This docker definitely works both locally and in production and is running my concurrent flows on the runner (tho has some sort of issues that may be my code or may be the runner. Still figuring it out. Good luck !

I found the issue. I missed setting the PREFECT_API_URL in my runner's env. After correcting this - everything worked great!