@Marvin In prefect 2.x, does the on_completed hook run when a task is cached?

Ah, the classic philosophical dilemma of whether the on_completed hook feels a sense of fulfillment when a task is cached. Fascinating. Now, are you indeed using Prefect 2.x? If so, I'll take a deep dive into the murky depths of Prefect documentation to find your answer.

@Marvin Prefect 2.x

lol i gotta make marvin less annoying about versions

The short and not-so-sweet answer: The documentation doesn't explicitly mention whether the

on_completed

hook is triggered when a task is cached. However, based on the excerpts, Prefect uses cache keys to determine if a task should reuse a cached state instead of running again. There's a lot of chatter about how caching works, but zero words about the intricacies of the

on_completed

hook in this scenario. For more in-depth reading, you can enjoy the full documentation here and here. If you're still curious (why do you even care?), I'd recommend trawling through some GitHub issues. If you're onboard, I can do that for you.

While you’re here… @Nate do you have an answer to this question?

yep one sec

Looking to run a progress update hook so I need it to run even if the task is cached

it appears not

In [1]: from prefect import flow, task

In [2]: @task(on_completion=[lambda *a, **kw: print("oi")], cache_key_fn=lambda *a: 'test')
   ...: def t(): ...

In [3]: @flow
   ...: def f():
   ...:     t()
   ...:     t()
   ...:

In [4]: f()
13:56:47.241 | INFO    | prefect.engine - Created flow run 'beneficial-panda' for flow 'f'
13:56:47.662 | INFO    | Flow run 'beneficial-panda' - Created task run 't-0' for task 't'
13:56:47.664 | INFO    | Flow run 'beneficial-panda' - Executing 't-0' immediately...
13:56:48.882 | INFO    | Task run 't-0' - Running hook '<lambda>' in response to entering state 'Completed'
oi
13:56:48.896 | INFO    | Task run 't-0' - Hook '<lambda>' finished running successfully
13:56:48.897 | INFO    | Task run 't-0' - Finished in state Completed()
13:56:49.078 | INFO    | Flow run 'beneficial-panda' - Created task run 't-1' for task 't'
13:56:49.080 | INFO    | Flow run 'beneficial-panda' - Executing 't-1' immediately...
13:56:49.234 | INFO    | Task run 't-1' - Finished in state Cached(type=COMPLETED)

Do you know of any workaround? It seems that this is enabled in prefect 3 https://docs.prefect.io/3.0/develop/manage-states#execute-code-on-state-changes

aha you seem right about that. hrm, would you be willing to open a

bug

on this specifically referencing 2.x?

Yep will do

thank you!!

Is it possible that this is new functionality for 3.0?

there’s a couple aspects to this • 3.x has a completely different engine, engine is responsible for running hooks • Cached is a type of Completed state, this is consistent between major versions i’m not sure exactly what the most “correct” behavior is yet, since i could reasonably argue either way • you should still be able to run the hook since the state transition into completed still takes place • cached should mean nothing happens so i think we can discuss on the issue which to codify in 3, but we may have to make this a setting if we want to backport it to 2.x otherwise it could cause unexpected breaking behavior for those counting on hooks not running on Cached tasks

Cached is a type of Completed state, this is consistent between major versions

This is what I drew me to the

on_completed

in the first place given that cached tasks log

Finished in state Cached(type=COMPLETED)

. My 2c is that at some point they must transition from being submitted to completing, and there should be a hook to capture this

> My 2c is that at some point they must transition from being submitted to completing, and there should be a hook to capture this 💯 we're on the same page! but we cannot change the default behavior at this point in 2.x so I think we'll need a setting do you have capacity/interest to contribute a PR for this? in 2.x, we'll need to add a new setting in settings.py like

PREFECT_RUN_ON_COMPLETION_HOOKS_ON_CACHED

or something and then find the place in

engine.py

(or code the engine calls) to put the new behavior behind the flag

Yep I can take a look

> Do you know if the task state for a cached task ever enters

running

? it does not appear so (just chucking some prints in

propose_state

)

removing as i put the prints in a confusing spot

which tracks for me • we try to run a task • server says, no we’ve already seen this hash key • proposed

Running

state is

REJECTED

-> take the recommended state from the server, which is

Cached

Got it sounds good then this would be a good place to start. Btw do you guys have any setup instructions for local dev, or how are you running these specific things?

Excellent

awesome! will take a look as soon as i can

Took another pass and tests are passing now!