https://prefect.io logo
#prefect-community
Title
# prefect-community
a

Anders Segerberg

04/12/2022, 4:20 PM
I have a workflow that writes a file to S3 I then have a flow-of-flows, which runs this workflow via
create_flow_run
, with the
idempotency_key
set to this file path. Running the flow the first time, will write this key. Running it a second time, won't -- as expected, because of the idempotency behavior. What I don't understand is that the second flow run has every task state set to 'success' -- but wouldn't I expect them to be 'cached' or something else, indicating that the flow isn't being re-ran?
k

Kevin Kho

04/12/2022, 4:23 PM
You have to re-run on the subflow page, because the
create_flow_run
task is just an API call with the same key that will return the already ran flow
a

Anders Segerberg

04/12/2022, 4:31 PM
I'm not sure how to get to the subflow page. Clicking on the "task" that amounts to the started flow run, and running 'restart' there, doesn't seem to indicate any cache behavior.
I use a
create_flow_run
followed by
wait_for_flow_run
k

Kevin Kho

04/12/2022, 4:32 PM
You can navigate to it in the UI. There will be a subflow run that triggered
a

Anders Segerberg

04/12/2022, 4:34 PM
Ok. So if I
restart
that flow, I would expect the state to be
cached
?
That is, the finished state?
a

Anna Geller

04/12/2022, 4:42 PM
I'm not sure how to get to the subflow page.
Technically, there are no subflows in Prefect 1.0. You just navigate to your flow page of the respective (child) flow
Clicking on the "task" that amounts to the started flow run, and running 'restart' there, doesn't seem to indicate any cache behavior.
Restarts are intended to be used when something fails, not to rerun any arbitrary task when needed.
So if I
restart
that flow, I would expect the state to be
cached
?
Caching is a separate feature and it applies to tasks, not to (sub)flows and it's used to avoid expensive recomputation. If you need more examples for the flow-of-flows, I recommend looking at some examples here and also create_flow_run examples.
Specifically, regarding restarting a flow-of-flows and using the idempotency key for that purpose, this page contains an example that may be particularly helpful in your use case
a

Anders Segerberg

04/12/2022, 5:35 PM
Ok. So when you set an idempotency key; if a subsequent run sees that it exists, that flow will simply not be run -- and it will return as a result, the result of the idempotent flow?
Which means that the 'state' of that sub-flow, in a subsequent flow run, is whatever the state of the first sub-flow run was
Which , in my case mentioned here, would be successfull; which is why I see that state
k

Kevin Kho

04/12/2022, 5:38 PM
Yes that’s right. So if you want a re-trigger of the subflow, you need something like caching the subflow tasks so they don’t re-run + a new idempotency key
4 Views