There are any resources/examples about a ETL pipel...
# ask-community
r
There are any resources/examples about a ETL pipeline for live data ingestion reading from a Queue and for each new event it creates a new flow that is executed by a central dask cluster with auto scaling? (actually this is a good use case for prefect? or should we stick with batch jobs?)
d
Hi Rodrigo! What sort of latency are you looking for? We don’t have docs for that specific example, but here are a couple of resources that may be useful to you! https://docs.prefect.io/api/latest/engine/executors.html#daskexecutor https://medium.com/the-prefect-blog/event-driven-workflows-with-aws-lambda-2ef9d8cc8f1a
Prefect can definitely ingest events from a queue, but we don’t support “steaming” in the Kafka sense (sub-millisecond latency, etc)
r
Just saw this now @Dylan Thank you a lot!!!!
👍 1