Question: we have a lot of cyclic jobs that run ev...
# ask-community
s
Question: we have a lot of cyclic jobs that run every 15 mins. We want to set up a self dependency such that they don’t run if the previous schedule fails or still running. We don’t even want to execute the flow so checking within the flow is not an option. How can I achieve this dependency Why? 1)We use k8s and if we have an issue say image pull secret expires the pods allocate cpu and go to pending state. That brings down the entire node pool or even cluster with 1000s of jobs going to pending state. Marvin didn’t give me a good answer 2) if the a run fails it’s likely every following run is going to fail creating 1000s of incidents in service now for example @Taylor Curran
👀 1
m
Hey @Sivanandha Rajadurai Two things come to mind here that would be useful deployment concurrency and automations. Specifically setting a deployment concurrency limit on relevant deployments allows you to define an a collision strategy i.e. what should happen when a run bumps up against a previous run for the same deployment that is still executing, you can enqueue the job so it executes as soon as a concurrency slot is available or you can cancel it, effectively skipping that run so you don't end up with a build up of jobs if the flow is consistently running longer than the schedule window between each run. For the failure part you could setup an automation that pauses the deployment/schedule if a run fails which would prevent future runs from occurring and subsequently send a notification to have someone investigate the failure, you may already be doing this part, until it was re-enabled either manually or through some other action
upvote 2
s
Thanks, is deployment concurrency available in Prefect 2.x? I cant seem to find it. Also can I pass the concurrency in prefect.yaml instead of command line?
I think i have answer. Thanks