https://prefect.io
Join Slack
I have a workflow that writes a file to S3 I then have a flow-of-flows, which runs this workflow via...
# ask-community
a
I have a workflow that writes a file to S3 I then have a flow-of-flows, which runs this workflow via 
create_flow_run
, with the 
idempotency_key
set to this file path. Running the flow the first time, will write this key. Running it a second time, won't -- as expected, because of the idempotency behavior. What I don't understand is that the second flow run has every task state set to 'success' -- but wouldn't I expect them to be 'cached' or something else, indicating that the flow isn't being re-ran?
k
You have to re-run on the subflow page, because the 
create_flow_run
task is just an API call with the same key that will return the already ran flow
a
I'm not sure how to get to the subflow page. Clicking on the "task" that amounts to the started flow run, and running 'restart' there, doesn't seem to indicate any cache behavior.
I use a 
create_flow_run
followed by 
wait_for_flow_run
k
You can navigate to it in the UI. There will be a subflow run that triggered
a
Ok. So if I 
restart
that flow, I would expect the state to be 
cached
?
That is, the finished state?
a
I'm not sure how to get to the subflow page.
Technically, there are no subflows in Prefect 1.0. You just navigate to your flow page of the respective (child) flow
Clicking on the "task" that amounts to the started flow run, and running 'restart' there, doesn't seem to indicate any cache behavior.
Restarts are intended to be used when something fails, not to rerun any arbitrary task when needed.
So if I 
restart
that flow, I would expect the state to be 
cached
?
Caching is a separate feature and it applies to tasks, not to (sub)flows and it's used to avoid expensive recomputation. If you need more examples for the flow-of-flows, I recommend looking at some examples here and also create_flow_run examples.
Specifically, regarding restarting a flow-of-flows and using the idempotency key for that purpose, this page contains an example that may be particularly helpful in your use case
a
Ok. So when you set an idempotency key; if a subsequent run sees that it exists, that flow will simply not be run -- and it will return as a result, the result of the idempotent flow?
Which means that the 'state' of that sub-flow, in a subsequent flow run, is whatever the state of the first sub-flow run was
Which , in my case mentioned here, would be successfull; which is why I see that state
k
Yes that’s right. So if you want a re-trigger of the subflow, you need something like caching the subflow tasks so they don’t re-run + a new idempotency key
6 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by