Hi there! My team is exploring Prefect as a workflow engine to support hundreds of data integrity checks on an internal survey management system containing tens of thousands of survey responses. We like Prefect because of its clean implementation, seemingly lower learning curve, and the ability to connect complex dependencies. I am tasked with building a prototype "flow" which can serve as an example that supports thousands of API calls, data models/checks, db reads and writes. My goal is to hit the ground running with the Prefect Core framework, using a threaded environment that can schedule tasks (i.e. API calls) in parallel, read and write to PG in bulk, and perform other various tasks such as existence checking, data integrity, etc. Coming from a Luigi background, a lot of these things are taken care of for me. Our biggest pain point with Luigi is its dependency management model + rigid existence checking, which can be a huge time suck as these checks are performed on 1 thread. I am seeking scalable granularity in this workflow. As I read through these docs, I am seeing your concept of
Executors
as well as the
DaskExecutor
object - which seems to be the proper choice. Now, when I start exploring this idea of
mapping
and connecting these task dependencies together, I get a little flustered without a more complex Prefect pipeline example... If it were possible, would you be able to point me to a larger scale example on GH or elsewhere; something that has multiple modules + a nicely defined project structure?