So with the Transform part of the ETL workflow, should there be 1 task for the transform that has multiple functions? or should there be multiple tasks?
ex. If
data
is a list of records/dicts, and I want to process them into a csv.
Copy code
with Flow('some flow') as flow:
data = readData(source)
transformedData = transformRecordsOneWay.map(data)
transformedData = transformRecordsSecondWay.map(transformedData)
transformedData = transformRecordsThirdWay.map(transformedData)
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
