is this the correct way to tackle the issue? How i can make sure that memory will not blow up as each of the load_data will save its result in memory, how can i control the number of parallel transform that can run ?
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
