Matt Delacour11/29/2022, 3:34 PM
Serina11/29/2022, 7:13 PM
task executes some SQL query, maybe you could edit it to retrieve 10k user IDs where some flag is set to false in the table, then you could have a final task in your flow that only runs if save_to_redshift task competes successfully, perhaps updating the original table to set the flag for those user IDs to true. And provided you have some schedule as well, maybe a statement in the flow like
returns some user IDs, execute the following tasks.
Kalise Richmond11/29/2022, 7:48 PM
in 10k, then run the transformation
over those chunks and finally once all those the chunks have run, upload the entire result to Redshift or save each chunk result to redshift?
Matt Delacour11/29/2022, 9:19 PM