How are people running de-duplication of flow runs based on unique parameters? I'm basically looking for a "distributed task lock/claim" using just Prefect. This is necessary for event-triggered flows, since Prefect does not guarantee exactly-once event delivery. Global concurrency limits are too restrictive since concurrent flow runs with different parameters are expected and OK. Tag-based concurrency limits must be pre-configured, whereas parameters are only known at runtime. There's the "distributed task claim" implementation through the transaction lock feature, but that requires deploying a self-managed redis instance. I'm surprised that this is not included in Prefect -- we already get a really feature-rich event system, task caching, and assets, all packaged in Prefect Cloud. Why not add a (ostensibly cheap) distributed task lock to the package? This must be a common use case.

hi @Nicolas Reichel - good question! i think the redis locking you're pointing out is orthogonal to what you're after, but i can see why something like this might be useful technically, you should be able to get what you're after by setting

idempotency_key

if you have a moment to check out a sketch here for how this could work in a first class way for automations (support for this could be ported to cloud): https://github.com/PrefectHQ/prefect/pull/21193/changes

Hi Nate! What a quick response, did you have this in the barrel already or did I just catch you at the right time? :D My understanding was that with an

idempotency_key

set on a transaction, you only deduplicate on a committed transaction (E.g. for a flow-level transaction, after at least one run has

Completed

), so you cannot prevent race conditions. Skimming the PR, it seems like a

TriggeredAction

has an

idempotency_key

as well, are you saying this functions as a claim/lock on the triggered action, e.g. is committed & checked (where? results store?) at the start of a triggered automation? And if I understand correctly, your PR is to enable parsing jinja templated event context (e.g. payload) into such a

idempotency_key

? If so, then yes this would solve my problem! Is there any documentation on this automation-level idempotency mechanism? I'd be really stoked if this could be added to Cloud as well ASAP. It would enable exactly-once event triggering based on event context, which is a huge feature upgrade for the platform. It's honestly a make-or-break feature for me. I'll happily attest this to a PM/PO if you think it could increase priority for release.