Hi All, I am trying to use Prefect caching at the task which works perfectly fine but I am having couple of issues: 1. When the same flow is run almost at the similar time which contains two task A & B. For caching to come effect the first flow has to run task A (then it cache it) and similarly when it completes task B (it would cache it). The issue comes when the same flow is run almost similar time when the tasks are not cached as the first trigger Flow might be still be processing task A and suddenly second Flow sees that task A is not cached and it would start even processing the same task A. 2. We do have some complex workflow that we are moving away from Celery without much rewriting. There are times when our two flows are different but they share some common task and I wanted to make sure that when these two different flow run at the same time the common task might end up running twice which is bit time consuming and cpu intensive. I am not sure how best to proceed with the above two issues.

Hi Vipul! This sounds like a good use case for task concurrency limits, which is a Cloud feature that allows you to limit the number of similar tasks running at one time. Check out this doc for more detail: https://docs.prefect.io/orchestration/concepts/task-concurrency-limiting.html

Thanks Chris but what I might have forgot to mention is that we only want the concurrency limit only when the input to task differ - Let say if Task A is called with multiple times with the same date as 09-Feb-2021 than we only want it to run once but if Task A is called multiple time with different date than we don’t want the concurrency limit