Hi folks - is it possible to restart a flow run but with a different context or run config ?

Marwan Sarieddine

11/15/2021, 10:27 PM

Hi folks - is it possible to restart a flow run but with a different context or run config ?

Marwan Sarieddine

11/15/2021, 10:27 PM

The use case is we specify our resource requirements (memory and cpu) via the flow’s context. Our flow run failed due to OOM error, so would like to re-run it but supply it with more resources by updating our context …

Kevin Kho

11/15/2021, 11:06 PM

Hey @Marwan Sarieddine, I don’t think so because restarts will use the same context. Maybe you could use caching though to preserve the old run results when you kick off a new flow run?

Marwan Sarieddine

11/15/2021, 11:10 PM

Hi @Kevin Kho thank you for your quick response I see - yes having caching enabled would have resolved this I suppose …

Marwan Sarieddine

11/15/2021, 11:11 PM

So there is no way to run a flow with an old flow run’s results without caching right ?

Marwan Sarieddine

11/15/2021, 11:13 PM

it sort of places a bit of a constraint - i.e. if the cache gets invalidated one can’t really do much to recover a failure

Kevin Kho

11/15/2021, 11:13 PM

You can make a temporary flow run and set the

target

for the file so that it would check for the existence of that file and then load the data

Marwan Sarieddine

11/15/2021, 11:14 PM

right

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.