https://prefect.io logo
Title
h

hakki cankaya

05/19/2020, 7:53 PM
Thanks @Zachary Hughes. Say if the host executor fails in mid execution or there is a server power outage. How can it survive and be resilient?
z

Zachary Hughes

05/19/2020, 7:56 PM
Ah, gotcha! I think I may have misunderstood your initial question then, @hakki cankaya. For recommendations on how to make your flows more resilient, I'd recommend this as a jumping-off point. https://docs.prefect.io/core/tutorial/04-handling-failure.html
h

hakki cankaya

05/19/2020, 8:05 PM
Thanks. Can I run prefect in a distributed way over a local server cluster that I have? say cluster of 2-4 servers?
z

Zachary Hughes

05/19/2020, 8:23 PM
Definitely! There may be some configuration involved depending on your local infrastructure. If you're looking for a distributed solution, Dask may be a good fit for you. https://docs.prefect.io/core/advanced_tutorials/dask-cluster.html#deployment-dask
h

hakki cankaya

05/19/2020, 9:05 PM
@Zachary Hughes Can i use my Spark cluster instead of Dask?
z

Zachary Hughes

05/19/2020, 9:24 PM
Good question! Prefect can orchestrate jobs on Spark, but Prefect flows cannot run directly on Spark. This is because we don't have a
SparkExecutor
(yet).
h

hakki cankaya

05/19/2020, 9:42 PM
Got it, thanks Zachary. Can you recommend a good simple example in docs or somewhere for an event based flow run, not scheduled?
z

Zachary Hughes

05/19/2020, 9:50 PM
If possible, could you provide a bit of context for your use case? Prefect doesn't support event-driven flows in a first class way at the moment, but a lot of our users have found patterns in Prefect that work just as well for them.
It's also worth noting that we're actively discussing how to best implement event-driven flows. @Laura Lorenz (she/her) has been soliciting community feedback-- I'm sure she'd love to hear a bit about what you're trying to do!
:upvote: 1
j

Jeremiah

05/19/2020, 10:23 PM
@hakki cankaya if it meets your needs, you can kick off a run at any time from the API (without a schedule); you could wire that up to any trigger you choose: here’s an example using AWS lambda https://medium.com/the-prefect-blog/event-driven-workflows-with-aws-lambda-2ef9d8cc8f1a
h

hakki cankaya

05/19/2020, 10:34 PM
I was more thinking of a Kafka message triggering a Prefect flow, or another message queue. Is there a recommended best practice? I guess @Jeremiah’s asynch API call is for one, if I understood correctly?
j

Jeremiah

05/19/2020, 10:35 PM
For the moment, an API call is recommended, but @Laura Lorenz (she/her) is looking into other approaches as Zachary mentioned
z

Zachary Hughes

05/19/2020, 10:35 PM
Correct! So in this case, you'd want something to watch for that Kafka message trigger, and whatever's watching would call
create_flow_run
in our API, creating a flow run!
h

hakki cankaya

05/19/2020, 10:45 PM
That makes sense. @Zachary Hughes What is your definition of first class way for event-driven flows?
z

Zachary Hughes

05/19/2020, 10:47 PM
We've had some users ask for flows that respond to events mid-run, etc. But for your use case, it sounds like being able to kick off a flow run in response to an event should do the trick!