Rodrigo Neves

11/12/2020, 3:51 PM
There are any resources/examples about a ETL pipeline for live data ingestion reading from a Queue and for each new event it creates a new flow that is executed by a central dask cluster with auto scaling? (actually this is a good use case for prefect? or should we stick with batch jobs?)


11/12/2020, 4:01 PM
Hi Rodrigo! What sort of latency are you looking for? We don’t have docs for that specific example, but here are a couple of resources that may be useful to you!
Prefect can definitely ingest events from a queue, but we don’t support “steaming” in the Kafka sense (sub-millisecond latency, etc)

Rodrigo Neves

11/17/2020, 11:16 AM
Just saw this now @Dylan Thank you a lot!!!!
👍 1