Does Prefect have something analogous to the Airflow Operator for starting, monitoring, and terminating AWS EMR clusters?

Hey @Josiah Berkebile, I just looked and doesn’t look like it. We’d gladly accept contributions for this though 🙂

I'm kinda looking around for an alternative worflow orchestrator that's better suited to streaming workflows. Prefect looks awesome, but I'm not sure it's quite there yet, honestly. I'd love to contribute, but I say that about many things. There's so many awesome projects around these days that could benefit from a few more contributors.

You are right that Prefect is not meant for streaming applications. There is an overhead with monitoring the state of each individual task (and flow).

I'm just beginning my investigation. But Cadence caught my eye. I'm also going to read-up on Shipyard, but I'm not sure how well either of these supports streaming workflows, either, atm

supporting some form of streaming has been a popular request for a while - while this doesn’t unblock you in the short term, we are working on some changes that will make this much more feasible in the future so it’s on our radar!

Keep up the good work

Yup! A common pattern is using a Lambda to hit the API to trigger a parametrized flow run