Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

<@UKNSNMUE6> Regarding state and metadata (e.g. log of task runs), you mentioned a while back that Prefect Cloud will have this setup by default. If we wanted to roll our own DB, how custom is this kind of setup and is there any Airflow-like equivalent to setting up your "metadata db" with a script?

Hi <@UL2CZP81F> - it’s a fair question; I’m not sure what you’re asking when you ask “how custom is this kind of setup”, but it’s certainly possible.  We don’t provide any method for doing so, as we believe Cloud is the “correct” abstraction for state storage and flow run orchestration

in the non-Cloud version of Prefect then, there is no persistence layer then?

persistence, specific to , seeing all historical runs

and we'd have to write the adapter to do that ourselves?

yup that is correct; the non-Cloud version of Prefect is only the engine which is designed  purely for running flows in a minimal way

well - i signed up :slightly_smiling_face:

otherwise, I’d like to ideally host this on GKE if we can

That is certainly possible - would you mind shooting me an email at <mailto:chris@prefect.io|chris@prefect.io>?  Maybe include a tiny bit about your use case and we can set up a 30 min chat!