Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

<https://docs.prefect.io/latest/guides/big-data/?h=big+data|Big Data with Prefect>

<https://github.com/PrefectHQ/prefect-recipes/tree/main/flows-advanced/parent-orchestrator|Orchestrator Worker Pattern>

^ good for breaking a large dataset up into smaller, more manageable chunks for downstream processes