Thanks @Zachary Hughes. Say if the host executor fails in mid execution or there is a server power outage. How can it survive and be resilient?

Ah, gotcha! I think I may have misunderstood your initial question then, @hakki cankaya. For recommendations on how to make your flows more resilient, I'd recommend this as a jumping-off point. https://docs.prefect.io/core/tutorial/04-handling-failure.html

Thanks. Can I run prefect in a distributed way over a local server cluster that I have? say cluster of 2-4 servers?

Definitely! There may be some configuration involved depending on your local infrastructure. If you're looking for a distributed solution, Dask may be a good fit for you. https://docs.prefect.io/core/advanced_tutorials/dask-cluster.html#deployment-dask

@Zachary Hughes Can i use my Spark cluster instead of Dask?

Good question! Prefect can orchestrate jobs on Spark, but Prefect flows cannot run directly on Spark. This is because we don't have a

SparkExecutor

(yet).

Got it, thanks Zachary. Can you recommend a good simple example in docs or somewhere for an event based flow run, not scheduled?

If possible, could you provide a bit of context for your use case? Prefect doesn't support event-driven flows in a first class way at the moment, but a lot of our users have found patterns in Prefect that work just as well for them.

@hakki cankaya if it meets your needs, you can kick off a run at any time from the API (without a schedule); you could wire that up to any trigger you choose: here’s an example using AWS lambda https://medium.com/the-prefect-blog/event-driven-workflows-with-aws-lambda-2ef9d8cc8f1a

I was more thinking of a Kafka message triggering a Prefect flow, or another message queue. Is there a recommended best practice? I guess @Jeremiah’s asynch API call is for one, if I understood correctly?

For the moment, an API call is recommended, but @Laura Lorenz (she/her) is looking into other approaches as Zachary mentioned

Correct! So in this case, you'd want something to watch for that Kafka message trigger, and whatever's watching would call

create_flow_run

in our API, creating a flow run!

That makes sense. @Zachary Hughes What is your definition of first class way for event-driven flows?

We've had some users ask for flows that respond to events mid-run, etc. But for your use case, it sounds like being able to kick off a flow run in response to an event should do the trick!