Hey All, just trying to workout what would be the best approach to solve the issue of task waiting for external system file to be available. To give a bit of context, I have lot of task that could wait for certain files to be available from external system If all the tasks are waiting for the files to be available than it might soon use up all the worker pool Was wondering if there is a way to retry those blocking task after 10 seconds so it does not hold the worker if it is not doing anything with Flow("Get Files") as file: get_file_from_system_a() get_file_from_system_b() @task def get_file_from_system_a(): while file_not_available: # Instead of holding one worker to wait for this file and end up worker starvation, # I was wondering if there is a way to retry this task after 10 seconds sleep(10) process_task()

Hey @Vipul! You could raise a retry signal - however it will be retried in process if the retry is in < 30 seconds i think

That is great even I was thinking on that line on raising RETRY

In the meantime, here’s a quick primer on Signals: https://docs.prefect.io/api/latest/engine/signals.html

Wonderful, thanks Dylan

You can set a

retry_delay=datetime.timedelta(minutes=1)

okie, thanks, let me try that

🧐

I tried this and it worked - RETRY(start_time=datetime.datetime.now() + datetime.timedelta(seconds=10))

That makes sense

Thanks Dylan for your help

Anytime!

@Vipul Would you be able to post what your final code looked like? I tried

_raise_ RETRY(start_time=datetime.datetime.now() + datetime.timedelta(seconds=10))

, but it is still triggering immediately.

Hey @Kelby Yes, that is what I did. My code snippet for retry was: raise RETRY(“Retrying”, start_time=datetime.datetime.now() + datetime.timedelta(seconds=10))

Finally figured it out. I needed to use

datetime.datetime.utcnow()

. Thanks for your help!

Aha great