https://prefect.io logo
Title
a

Alderson

06/24/2021, 12:56 PM
Hey, first question in here - how do I get started if I'd like to deploy exclusively on docker? Does it really require python on host machine to run? Would prefer a simple docker-compose.yml to start and not have to install python if possible. Thanks!
k

Kevin Kho

06/24/2021, 1:03 PM
Hey @Alderson, I answered you in the introductions channel but we can keep the conversation going here. Yes you need Python. If you have a container that you already want to run, you can use the StartContainer task with PRefect.
z

Zanie

06/24/2021, 2:42 PM
Hey @Alderson -- if you're just trying to deploy Prefect Server without Python on the host machine you can generate the compose yaml on another machine first.
prefect server config > docker-compose.yaml
prefect server config
takes roughly the same settings as
prefect server start
If you're just getting started though, it's almost always easier to use the free Cloud tier first then switch to Server if you decide you need your own deployment.
a

Alderson

06/24/2021, 3:52 PM
oh, that's wonderful, thanks for that! I'll play around with that and see how it goes
Could you make it an FR to have docker-compose.yml available on the repo?
z

Zanie

06/24/2021, 3:53 PM
The base docker-compose file is available in the repo, but it's filled with settings from the config and the CLI which is why we provide a tool to generate one as well
a

Alderson

06/24/2021, 3:58 PM
If you're just getting started though, it's almost always easier to use the free Cloud tier first then switch to Server if you decide you need your own deployment.
Is there any difference between the cloud and the self hosted one as it stands currently? I understand UI was recently open sourced?
z

Zanie

06/24/2021, 4:33 PM
The UI has been open source for a while now. There are a few differences, but they're mostly auth related or non-beginner feature related like if you want automated notifications or global enforcement of limits like concurrent flow runs
a

Alderson

06/24/2021, 4:55 PM
Thanks, and if I understand this correctly, coming from a bit of airflow background where tasks are run by the same machine hosting the web ui, in prefect, the ui and scheduler is one service on one host (usually the cloud), and the actual tasks would be on a separate machine, correct? If this machine is offline, then the tasks do not get executed, yes? And prefect does not run it, only manage the execution. Am I understanding this correctly?
z

Zanie

06/24/2021, 4:55 PM
Yep!
You run your "agent" on a separate machine which polls the "scheduler" for ready runs then deploys runs to the machine it is on (local/docker agents) or to other infrastructure (ecs/kubernetes agents)
If you're running Prefect Server, it could be the same machine -- it's just frequently separate.
a

Alderson

06/24/2021, 5:05 PM
The Docker agent executes flow runs in individual Docker containers.
And so, let's say we have 5 flows to run, then
prefect agent docker start
will create 5 containers, one for each flow?
z

Zanie

06/24/2021, 5:06 PM
Yep!
It creates the containers in parallel as well -- once the agent deploys the flow it generally cedes responsibility for it
a

Alderson

06/24/2021, 5:06 PM
Btw michael, thanks a whole bunch for your replies, it makes the onboarding a whole lot less headache-inducing! 😄
cedes responsibility to the server/cloud?
so the cloud can stop the docker container when the flow is stopped?
z

Zanie

06/24/2021, 5:08 PM
No problem!
Cedes responsibility as in it does not usually require the agent to continue running for the flow run to succeed (I think a local agent is an exception since they are subprocesses) -- the agent is designed to be lightweight
If the flow is cancelled via the backend, the flow runner in the container will detect it and exit the container.
a

Alderson

06/24/2021, 5:12 PM
it does not usually require the agent to continue running for the flow run to succeed
Not too sure I understand this very well - if the agent is offline, how does the flow run?
z

Zanie

06/24/2021, 5:23 PM
The agent is required to start the flow but not for its continued operation
i.e. Agent is running -> Flow run is ready -> Agent deploys flow run as a K8s job -> Agent is killed -> Flow continues running