Hello there What s the best way to re run a task automatical

Question

Hello there What s the best way to re run a task automatically My use case I m fetching a lots of data but I can t put it all in memory so I need to re run the task until I ve access all the data Thank you

Kyle Moon-Wright · Answer

Hey < Nuno> Not sure if this will meet your specifications but I d look into that checks for a predetermined threshold and reruns a task until that condition is met

Kyle Moon-Wright · Answer

<https docs prefect io core advanced tutorials task looping html introducing task looping|Here s an example> of a LOOP signal in action to give you a better idea of how you can utilize this signal

Nuno · Answer

Hello Kyle Thanks for the reply I had already check the LOOP signal the problem is that it does everything in the same task process and it doesn t create a new one Which would be necessary since I don t want to have a process filling all the available memory

Kyle Moon-Wright · Answer

Gotcha that is a pickle Even dynamically creating tasks by mapping over your dataset wouldn t work because of the memory limitations thinking face

Nuno · Answer

anyway for now I ve found a workaround Basically I m going to have a post process task that verifies if the data i m processing is equal to my FETCH LIMIT if so than I programmatically tell to run another flow with a new cursor

Kyle Moon-Wright · Answer

nice this would be to kick off a flowrun of the same flow

Nuno · Answer

yes exactly

Kyle Moon-Wright · Answer

Awesome Great idea

Nuno · Answer

Thanks slightly smiling face

Nuno · Answer

Anyway thanks for your reply once more Maybe in future prefect could build some similar logic idk

Kyle Moon-Wright · Answer

Noted I like the idea for sure and I m glad you have something in place for the time being smile