Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi, newb here trying to find a replacement for airflow. What's the recommended infrastructure to deploy prefect core in production? Looking for minimum CPU/memory. The documentation only mentions a simple pip installation, no real discussion on dimensioning, external dependencies like databases, etc. Is it really that simple? Then why <http://prefect.io|prefect.io> cloud?

Hi <@U014YLBG8JW>.  Prefect does not have recommended infrastructure.  It really just depends on your Flow's  requirements.  Some purposes of prefect cloud are to allow users to monitor flows in real time, re-run a flow from a the last succesful task completed. and kick of scheduled flows.
The general steps for deploying a flow in the cloud are (1) Decide on the executor  (2) Add Storage (3) Register the flow with cloud (4)) Run  agent.

<https://docs.prefect.io/orchestration/tutorial/first.html#write-flow>

Hi <@UNSQ0T2V9> thanks for replying. I finally figured out how to use prefect  server start to spawn a bunch of docker containers. How can I contribute to the documentation, as it seems to me it is quite poor in this aspect? There are a lot of dependencies to install for example...

Now however, i have a bunch of services running, but I can see several errors happened during startup (first execution of prefect server start):

```graphql_1    | Could not upgrade the database!
graphql_1    | Error: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "pg_extension_name_index"```
The prefect UI complains about not being able to connect to the graphql API

ok i figured out that the client is trying to connect to 127.0.0.1, so i needed to create a ssh tunnel to the server, this does not seem to be a production-ready deployment recipe (sigh)

ok, it seems obvious from further reading and comparing the features of prefect-core and prefect cloud that it probably is better to go for the cloud version for any serious production ambitions

Hi <@U014YLBG8JW>, Prefect Server *should* be easier to start than your experience, so I’m sorry about that - however it is indeed not what we recommend for production use. It’s great as a basic workflow management system, but our goal with it was to deliver an orchestration system that could run as a single “docker compose application” and  (as you alluded), getting proper scale and infrastructure robustness is challenging.

We’d invite you to look at Prefect Cloud instead for a production use case, including the free version. With Prefect Cloud, all orchestration is managed by us but your code / data remain private and execute on your infrastructure.

<@U014YLBG8JW> i recently went through the process of spinning up Prefect server in kubernetes and it was significantly more complex than running airflow in production, at least for the intial spinup. Airflow has the webserver and the scheduler and the db, pretty straightforward. Prefect has more moving parts and how they work together is more opaque, but the overall benefits of Prefect for our DS team made it worth it.