Hello all, I'm quite new to Prefect and I'm looking for the best way to trigger a task based on external event. In my case the external event is a new S3 partition available and ready to be processed. To achieve that I've tried to rely on RETRY signal. Basically I defined a task in charge of checking the partition and raising the following signal in case it's not available yet:

raise RETRY("Partition not available yet, retrying in 1 hour.", start_time=pendulum.now().add(hours=1))

However it looks like this could block the other running tasks because the runner is not released. Now let's imagine we only rely on a single local agent, with maximum 3 tasks in parallel, in case some S3 partitions are late for any reason, it could quickly block all of the others tasks. Do you have any better way to achieve this ?

Can you share your full flow code? this would block other tasks only if downstream tasks depend on it, and if you raise RETRY, this task run keeps running until all retries are finished and you are done with this task. Raising a RETRY signal is a valid option, even though I usually see it being used with smaller intervals, say a couple of seconds or minutes. If you need to wait for a full hour, perhaps it makes sense to end a flow run and create a new independent separate flow run in 1 hour? Usually, in such use cases, I would recommend AWS Lambda but I don't think this would work with an on-prem Server setup.

Oh sorry, it looks like it's only an issue happening when retry time is < 10mn (cf this issue). I initially performed my test with a 5mn retry, but now using 1h it seems to work well according to latest tests. Indeed as you said only downstream tasks are waiting for it, but it's expected. Concerning the RETRY usage, I prefer still relying on a retry mechanism because I would like to couple the partition processed and the Prefect task, meaning: • a task processes a dedicated partition instead of • a task processes all of the new available partitions Thanks a lot for your answer and reactivity 🙂

Yup, it sounds like the right solution for the use case when downstream tasks need to process the same partition. Glad you got it working!