There are any resources/examples about a ETL pipeline for live data ingestion reading from a Queue and for each new event it creates a new flow that is executed by a central dask cluster with auto scaling? (actually this is a good use case for prefect? or should we stick with batch jobs?)

Hi Rodrigo! What sort of latency are you looking for? We don’t have docs for that specific example, but here are a couple of resources that may be useful to you! https://docs.prefect.io/api/latest/engine/executors.html#daskexecutor https://medium.com/the-prefect-blog/event-driven-workflows-with-aws-lambda-2ef9d8cc8f1a

Just saw this now @Dylan Thank you a lot!!!!