Hi, newb here trying to find a replacement for airflow. What's the recommended infrastructure to deploy prefect core in production? Looking for minimum CPU/memory. The documentation only mentions a simple pip installation, no real discussion on dimensioning, external dependencies like databases, etc. Is it really that simple? Then why prefect.io cloud?

Hi @Luis Muniz. Prefect does not have recommended infrastructure. It really just depends on your Flow's requirements. Some purposes of prefect cloud are to allow users to monitor flows in real time, re-run a flow from a the last succesful task completed. and kick of scheduled flows. The general steps for deploying a flow in the cloud are (1) Decide on the executor (2) Add Storage (3) Register the flow with cloud (4)) Run agent. https://docs.prefect.io/orchestration/tutorial/first.html#write-flow

Hi @itay livni thanks for replying. I finally figured out how to use prefect server start to spawn a bunch of docker containers. How can I contribute to the documentation, as it seems to me it is quite poor in this aspect? There are a lot of dependencies to install for example... Now however, i have a bunch of services running, but I can see several errors happened during startup (first execution of prefect server start):

graphql_1    | Could not upgrade the database!
graphql_1    | Error: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "pg_extension_name_index"

The prefect UI complains about not being able to connect to the graphql API

Hi @Luis Muniz, Prefect Server should be easier to start than your experience, so I’m sorry about that - however it is indeed not what we recommend for production use. It’s great as a basic workflow management system, but our goal with it was to deliver an orchestration system that could run as a single “docker compose application” and (as you alluded), getting proper scale and infrastructure robustness is challenging. We’d invite you to look at Prefect Cloud instead for a production use case, including the free version. With Prefect Cloud, all orchestration is managed by us but your code / data remain private and execute on your infrastructure.

@Luis Muniz i recently went through the process of spinning up Prefect server in kubernetes and it was significantly more complex than running airflow in production, at least for the intial spinup. Airflow has the webserver and the scheduler and the db, pretty straightforward. Prefect has more moving parts and how they work together is more opaque, but the overall benefits of Prefect for our DS team made it worth it.

thanks! @Thomas La Piana