Matt Delacour
11/29/2022, 3:34 PMSerina
11/29/2022, 7:13 PMget_user_ids
task executes some SQL query, maybe you could edit it to retrieve 10k user IDs where some flag is set to false in the table, then you could have a final task in your flow that only runs if save_to_redshift task competes successfully, perhaps updating the original table to set the flag for those user IDs to true.
And provided you have some schedule as well, maybe a statement in the flow like while get_user_ids()
returns some user IDs, execute the following tasks.Kalise Richmond
11/29/2022, 7:48 PMget_user_ids()
in 10k, then run the transformation get_third_party_api_info()
over those chunks and finally once all those the chunks have run, upload the entire result to Redshift or save each chunk result to redshift?Matt Delacour
11/29/2022, 9:19 PM