https://prefect.io logo
o

Ofir

07/21/2023, 7:00 PM
Is Prefect applicable for stateful transactional workflows?
p

Peter

07/21/2023, 7:09 PM
It's python - so if you can do the work for your 'stateful transactional workflow' in Python - you MIGHT be able to do it with prefect.
n

Nate

07/21/2023, 7:17 PM
hi @Ofir prefect certainly helps with the stateful part of workflows, I'd be curious what you have in mind as far as the transactional part goes
o

Ofir

07/22/2023, 11:20 AM
@Peter @Nate thanks for the reply! I’ll provide an example. A simple example of Amazon, e.g. an E-commerce scenario could be an order placement operation, which could consist of several stages: 1. Validate order 2. Update inventory 3. Deduct payment 4. etc. If any of these steps fails, the operation should be undone right? What happens if the payment in step (3) was unsuccessful? The item that was pulled out from the inventory needs to be put back in or in other words, we need to rollback / we need a compensating transaction. Is Prefect suited for these types of workflows (which Zeebe handles) or is Prefect primarily for stateless DAGs for Data Science / Data Engineering tasks?
An order from Amazon goes through a state machine: pending -> placed -> paid -> shipped -> etc.
@Nate your thoughts? Motivation / context - the Saga design pattern for distributed microservices and transactions in E-commerce for example https://microservices.io/patterns/data/saga.html
p

Peter

07/24/2023, 1:05 PM
Not necessarily stateless DAGs - but it looks like your use case might be better served by serverless microservices - not data pipeline tools.
🙌 1
o

Ofir

07/24/2023, 8:12 PM
Thanks Peter I appreciate that.
n

Nate

07/24/2023, 8:13 PM
yeah, I believe you'd have to implement your own rollback behaviour using state hooks or something - your use case is not my area of expertise! apologies