Dylan
02/25/2022, 10:46 PMversion: '3'
services:
python-repl:
image: prefect
entrypoint: "/bin/bash"
volumes:
- "workspace:/home/workspace/"
ports:
- "4200:4200"
prefect-backend-only:
image: prefect
command: prefect backend server && prefect agent local start && prefect server start
volumes:
- "workspace:/home/workspace/prefect-server"
prefect-full-stack:
image: prefect
command: prefect server start
volumes:
- "workspace:/home/workspace/prefect-server"
volumes:
workspace:
the dockerfile I used to build the prefect
image above is just a python image I’ve installed conda and prefect on. I can run the prefect backend server
command just fine but it blows up on prefect agent local start
with
requests.exceptions.ConnectionError: HTTPConnectionPool(host='apollo', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4004761130>: Failed to establish a new connection: [Errno -2] Name or service not known'))
I could just be missing some docker-compose pieces, and ideally I could re-use the docker-compose pieces your development team uses for the local dev stack.Kevin Kho
02/25/2022, 10:58 PMprefect agent .... start
it starts the agent that then polls the API of Prefect Server and listens to flows. You get an error because your start the agent before spinning up your server with prefect server start
. At the very minimum, you likely need to reverse those commands.
You should also be aware Prefect Cloud has 20k free task runs every month no credit card needed so you don’t have to spin up server yourselfDylan
02/25/2022, 11:06 PMroot@4143c36edf59:/# prefect backend server
Backend switched to server
root@4143c36edf59:/# prefect server start
Exception caught; killing services (press ctrl-C to force)
...
...
FileNotFoundError: [Errno 2] No such file or directory: 'docker-compose'
prefect backend server
is meant to be run on an OS with the prefect
pacakge intstalled, as well as a docker agent, and that the prefect backend server is that collection of docker-compose services in the GH source?Kevin Kho
02/25/2022, 11:10 PMprefect server start --expose
and that should be it for the most part. the agent does not necessarily have to be on the same machine as the server but it can be alsoprefect server start
spins up the UI, graphql api, database, etc as docker containersDylan
02/25/2022, 11:12 PMKevin Kho
02/25/2022, 11:17 PMprefect server start
should spin up the servicesprefect server start
does the docker compose for youDylan
02/25/2022, 11:21 PMKevin Kho
02/25/2022, 11:27 PMDylan
02/25/2022, 11:30 PMtask
for billing purposes? Would that be any successful invocation of a python function using the @task
decorator? Can you also provide some context around when and why someone would decide to run FOSS prefect and not use your Cloud?Kevin Kho
02/28/2022, 6:32 PMScheduled -> Submitted -> Running -> Success
so we charge for the observability into that TaskDylan
02/28/2022, 6:32 PMKevin Kho
02/28/2022, 6:32 PMDylan
02/28/2022, 6:40 PMKevin Kho
02/28/2022, 6:43 PMDylan
02/28/2022, 6:44 PMKevin Kho
02/28/2022, 6:46 PMDylan
02/28/2022, 6:48 PMprefect
cli and generating the docker-compose yml, or are there docker-compose & helmchart templates for the agent stand-alone from the server stack compose generator?Kevin Kho
02/28/2022, 6:54 PM