Hey all! A question I couldn't find addressed in the docs: is there a way to run a flow from a particular "entry point"? E.g. if you have a flow with sequential tasks A -> B -> C -> D, could you invoke the flow from task B or C? A concrete use case is a data processing pipeline where task A is time-consuming and most of the time doesn't need to be run.

Hi @David Norrish -- Prefect has a caching mechanism for exactly this use case! You can indicate that a task's final state should be cached for a specific amount of time. This let's you skip expensive steps for every run.

Ah thanks! That's good to know

The cache is invalidated if it expires or if certain conditions aren’t met (like parameters change). However it must be provided as an initial

CachedState

for the task in question when you run the flow, so the choice of how to persist and whether to provide it is up to you. If you want to forcibly invalidate it, you could simply run your flow with no initial states provided, and then the task would run as normal.

Thanks a bunch @Jeremiah! I just found the relevant section of the docs (https://docs.prefect.io/guide/core_concepts/execution.html#caching), so will have a play and see if I can get it working as needed. I hadn't come across the concept of an "initial state". So if I understand correctly, one would execute the

Flow

, and hold onto the

CachedState

. Then to run a second time and skip the cached steps, you would pass any `CachedState`s as a list to the

tasks

parameter of

flow.run()

?

Let me defer to @Chris White for a recommendation there. They are JSON serializable by design but this starts to involve result handlers to serialize the result itself, and as I’m afk I may not be able to fully explain

Thanks!

Hey @David Norrish! so a few things: - to specify a set of initial states, you can pass a dictionary of

{task: state}

to

flow.run

under the

task_states

keyword - this kwarg is passed down to the flow runner itself and is documented here: https://docs.prefect.io/api/unreleased/engine/flow_runner.html#prefect-engine-flow-runner-flowrunner-run

Thanks a bunch @Chris White 🙂 That's super handy to have explained. We're still in the "poking around and trying it out" stage, so not on Prefect Cloud yet, but that's definitely a possibility for down the line! Shall see how I go with this approach

yea that sounds good, and definitely let us know if the patterns outlined here don’t fit your use case!

Will do!

@Marvin archive “How to run a Flow starting from a particular task?”