a

    Alderson

    1 year ago
    Hey, first question in here - how do I get started if I'd like to deploy exclusively on docker? Does it really require python on host machine to run? Would prefer a simple docker-compose.yml to start and not have to install python if possible. Thanks!
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Alderson, I answered you in the introductions channel but we can keep the conversation going here. Yes you need Python. If you have a container that you already want to run, you can use the StartContainer task with PRefect.
    Michael Adkins

    Michael Adkins

    1 year ago
    Hey @Alderson -- if you're just trying to deploy Prefect Server without Python on the host machine you can generate the compose yaml on another machine first.
    prefect server config > docker-compose.yaml
    prefect server config
    takes roughly the same settings as
    prefect server start
    If you're just getting started though, it's almost always easier to use the free Cloud tier first then switch to Server if you decide you need your own deployment.
    a

    Alderson

    1 year ago
    oh, that's wonderful, thanks for that! I'll play around with that and see how it goes
    Could you make it an FR to have docker-compose.yml available on the repo?
    Michael Adkins

    Michael Adkins

    1 year ago
    The base docker-compose file is available in the repo, but it's filled with settings from the config and the CLI which is why we provide a tool to generate one as well
    a

    Alderson

    1 year ago
    If you're just getting started though, it's almost always easier to use the free Cloud tier first then switch to Server if you decide you need your own deployment.
    Is there any difference between the cloud and the self hosted one as it stands currently? I understand UI was recently open sourced?
    Michael Adkins

    Michael Adkins

    1 year ago
    The UI has been open source for a while now. There are a few differences, but they're mostly auth related or non-beginner feature related like if you want automated notifications or global enforcement of limits like concurrent flow runs
    a

    Alderson

    1 year ago
    Thanks, and if I understand this correctly, coming from a bit of airflow background where tasks are run by the same machine hosting the web ui, in prefect, the ui and scheduler is one service on one host (usually the cloud), and the actual tasks would be on a separate machine, correct? If this machine is offline, then the tasks do not get executed, yes? And prefect does not run it, only manage the execution. Am I understanding this correctly?
    Michael Adkins

    Michael Adkins

    1 year ago
    Yep!
    You run your "agent" on a separate machine which polls the "scheduler" for ready runs then deploys runs to the machine it is on (local/docker agents) or to other infrastructure (ecs/kubernetes agents)
    If you're running Prefect Server, it could be the same machine -- it's just frequently separate.
    a

    Alderson

    1 year ago
    The Docker agent executes flow runs in individual Docker containers.
    And so, let's say we have 5 flows to run, then
    prefect agent docker start
    will create 5 containers, one for each flow?
    Michael Adkins

    Michael Adkins

    1 year ago
    Yep!
    It creates the containers in parallel as well -- once the agent deploys the flow it generally cedes responsibility for it
    a

    Alderson

    1 year ago
    Btw michael, thanks a whole bunch for your replies, it makes the onboarding a whole lot less headache-inducing! 😄
    cedes responsibility to the server/cloud?
    so the cloud can stop the docker container when the flow is stopped?
    Michael Adkins

    Michael Adkins

    1 year ago
    No problem!
    Cedes responsibility as in it does not usually require the agent to continue running for the flow run to succeed (I think a local agent is an exception since they are subprocesses) -- the agent is designed to be lightweight
    If the flow is cancelled via the backend, the flow runner in the container will detect it and exit the container.
    a

    Alderson

    1 year ago
    it does not usually require the agent to continue running for the flow run to succeed
    Not too sure I understand this very well - if the agent is offline, how does the flow run?
    Michael Adkins

    Michael Adkins

    1 year ago
    The agent is required to start the flow but not for its continued operation
    i.e. Agent is running -> Flow run is ready -> Agent deploys flow run as a K8s job -> Agent is killed -> Flow continues running