Hello there. What’s the best way to re-run a task,...
# prefect-communityn
Nuno
10/26/2020, 4:39 PMHello there. What’s the best way to re-run a task, automatically? My use case: I’m fetching a lots of data, but I can’t put it all in memory, so I need to re run the task until I’ve access all the data. Thank you
k
Kyle Moon-Wright
10/26/2020, 5:40 PMHey @Nuno,
Not sure if this will meet your specifications, but I'd look into LOOP signals that checks for a predetermined threshold and reruns a task until that condition is met.
Kyle Moon-Wright
10/26/2020, 5:42 PMHere's an example of a LOOP signal in action to give you a better idea of how you can utilize this signal.
n
Nuno
10/26/2020, 5:59 PMHello Kyle: Thanks for the reply! I had already check the LOOP signal, the problem is that it does everything in the same task process, and it doesn’t create a new one. Which would be necessary, since I don’t want to have a process filling all the available memory.
k
Kyle Moon-Wright
10/26/2020, 6:09 PMGotcha, that is a pickle. Even dynamically creating tasks by mapping over your dataset wouldn't work because of the memory limitations. 🤔
n
Nuno
10/26/2020, 6:11 PManyway, for now I’ve found a workaround… Basically I’m going to have a post process task, that verifies if the data i’m processing is equal to my FETCH LIMIT… if so, than I programmatically tell to run another flow, with a new cursor.
🚀 1
k
Kyle Moon-Wright
10/26/2020, 6:13 PMnice! this would be to kick off a flowrun of the same flow?
n
Nuno
10/26/2020, 6:13 PMyes, exactly
🎉 1
k
Kyle Moon-Wright
10/26/2020, 6:14 PMAwesome! Great idea.
n
Nuno
10/26/2020, 6:14 PMThanks 🙂
Nuno
10/26/2020, 6:15 PMAnyway, thanks for your reply once more. Maybe in future, prefect could build some similar logic… idk
k
Kyle Moon-Wright
10/26/2020, 6:17 PMNoted! I like the idea for sure, and I'm glad you have something in place for the time being 😄 .
👍 1