Channels
pacc-may-31-2023
prefect-ai
pacc-clearcover-june-12-2023
marvin-in-the-wild
data-ecosystem
geo-israel
geo-japan
prefect-cloud
ppcc-may-16-2023
prefect-azure
prefect-docker
prefect-recipes
gratitude
geo-nyc
geo-bay-area
geo-boston
geo-london
geo-dc
geo-chicago
geo-berlin
geo-texas
geo-seattle
geo-colorado
prefect-community
data-tricks-and-tips
prefect-aws
prefect-gcp
introductions
find-a-prefect-job
prefect-dbt
random
events
ask-marvin
show-us-what-you-got
prefect-getting-started
prefect-integrations
prefect-contributors
best-practices-coordination-plane
announcements
prefect-server
prefect-ui
prefect-kubernetes
Title
a
Alderson
06/24/2021, 12:56 PMHey, first question in here - how do I get started if I'd like to deploy exclusively on docker? Does it really require python on host machine to run? Would prefer a simple docker-compose.yml to start and not have to install python if possible. Thanks!
k
Kevin Kho
06/24/2021, 1:03 PMHey @Alderson, I answered you in the introductions channel but we can keep the conversation going here. Yes you need Python. If you have a container that you already want to run, you can use the StartContainer task with PRefect.
z
Zanie
06/24/2021, 2:42 PMHey @Alderson -- if you're just trying to deploy Prefect Server without Python on the host machine you can generate the compose yaml on another machine first.
prefect server config > docker-compose.yamlprefect server configprefect server startIf you're just getting started though, it's almost always easier to use the free Cloud tier first then switch to Server if you decide you need your own deployment.
a
Alderson
06/24/2021, 3:52 PMoh, that's wonderful, thanks for that! I'll play around with that and see how it goes
Could you make it an FR to have docker-compose.yml available on the repo?
z
Zanie
06/24/2021, 3:53 PMThe base docker-compose file is available in the repo, but it's filled with settings from the config and the CLI which is why we provide a tool to generate one as well
a
Alderson
06/24/2021, 3:58 PMIf you're just getting started though, it's almost always easier to use the free Cloud tier first then switch to Server if you decide you need your own deployment.Is there any difference between the cloud and the self hosted one as it stands currently? I understand UI was recently open sourced?
z
Zanie
06/24/2021, 4:33 PMThe UI has been open source for a while now. There are a few differences, but they're mostly auth related or non-beginner feature related like if you want automated notifications or global enforcement of limits like concurrent flow runs
a
Alderson
06/24/2021, 4:55 PMThanks, and if I understand this correctly, coming from a bit of airflow background where tasks are run by the same machine hosting the web ui, in prefect, the ui and scheduler is one service on one host (usually the cloud), and the actual tasks would be on a separate machine, correct? If this machine is offline, then the tasks do not get executed, yes? And prefect does not run it, only manage the execution. Am I understanding this correctly?
z
Zanie
06/24/2021, 4:55 PMYep!
You run your "agent" on a separate machine which polls the "scheduler" for ready runs then deploys runs to the machine it is on (local/docker agents) or to other infrastructure (ecs/kubernetes agents)
If you're running Prefect Server, it could be the same machine -- it's just frequently separate.
a
Alderson
06/24/2021, 5:05 PMThe Docker agent executes flow runs in individual Docker containers.And so, let's say we have 5 flows to run, then
prefect agent docker startz
Zanie
06/24/2021, 5:06 PMYep!
It creates the containers in parallel as well -- once the agent deploys the flow it generally cedes responsibility for it
a
Alderson
06/24/2021, 5:06 PMBtw michael, thanks a whole bunch for your replies, it makes the onboarding a whole lot less headache-inducing! 😄
cedes responsibility to the server/cloud?
so the cloud can stop the docker container when the flow is stopped?
z
Zanie
06/24/2021, 5:08 PMNo problem!
Cedes responsibility as in it does not usually require the agent to continue running for the flow run to succeed (I think a local agent is an exception since they are subprocesses) -- the agent is designed to be lightweight
If the flow is cancelled via the backend, the flow runner in the container will detect it and exit the container.
a
Alderson
06/24/2021, 5:12 PMit does not usually require the agent to continue running for the flow run to succeedNot too sure I understand this very well - if the agent is offline, how does the flow run?
z
Zanie
06/24/2021, 5:23 PMThe agent is required to start the flow but not for its continued operation
i.e. Agent is running -> Flow run is ready -> Agent deploys flow run as a K8s job -> Agent is killed -> Flow continues running