Hello, I got interested in Prefect as I was investigating Python based orchestration engine. First off, can we use MYSQL as the underlying DB instead of Postgres. Our use case is that we want to define a flow with some tasks. And that flow's trigger should be event driven (say a message in queue). We can have potentially 100's of thousands of those triggers in a day, and the tasks are short tasks. But the individual tasks involve calling some api, but we get the response asynchronously. So the next task can only proceed after previous task was successful, but we need another trigger to inform the flow that previous task was successful. I wonder if Prefect can be used to solve this particular use case ? Also we'll use a self-hosted env, if that makes any difference.

For the db question when spinning up the Core server it defaults to using Postgres so if you wanted to use MySQL you may have to do some extra configuration in order to get it to work. The process uses alembic for migrations so if that works against MySQL I don’t think it would be unreasonable. (I only have experience using it with Postgres so I’m not too sure on the trade offs) Prefect flows can be triggered based on events! Calling the API to trigger a flow run would be how you would do it. There is also a task in the task library for starting new flow runs (or you can roll your own 🙂) which could be used to trigger a new run based on some upstream state.

Hi Jayakar! Quick addendum to Josh's great answer: You'll likely have a rough time if you're trying to use MySQL as the underlying persistence layer for Prefect Server. The reason for this is that Hasura is currently only compatible with Postgres. That said, it sounds like Hasura's working on MySQL support! https://www.producthunt.com/upcoming/hasura-graphql-engine-on-mysql

Thank you for your answer. I didn't quite get the asynchronous task handling. Flow -> T1, T2, T3. Flowrun starts. T1 is done. T2 calls api -> gets response. -> T2 goes to PAUSE state. Is there a way to mark T2 as done using an API from outside, so that T3 can start automatically.

Sure thing. Prefect Server and Cloud have fully-fledged GraphQL APIs, so you should be able to use the

set_task_run_state

mutation to change task run states however you'd like.

Ok, thanks. I'll play around with this and see how it goes. And yes, I'll consider the Postgres suggestion. Since it's Hasura, I guess almost everything will be available as API.