Hi When flow task is retrying in say 1 minute it s scheduled

Question

Hi When flow task is retrying in say 1 minute it s scheduled start time in context is +1 minute from the failed one I m using scheduled start time kind of how I would use DagRun s timestamp key in Airflow in an SQL template so this behaviour isn t helpful Am I doing it wrong Can you get the original task s metadata context

Jeremiah · Answer

Hi < Mikhail Akimov> Good question To be clear you want the start time or scheduled start time of the overall flow run itself not the task run right

Jeremiah · Answer

I m looking in the code and realizing that we store both under the same context key `scheduled start time` so I believe the task value overwrites the flow value

Mikhail Akimov · Answer

Oh so the original flow start time kind of should be there but overwritten

Jeremiah · Answer

Yep and I m going to PR a fix right now

Mikhail Akimov · Answer

Super nice thank you

Jeremiah · Answer

Thank you for pointing this out

Jeremiah · Answer

Actually I apologize we don t set the `scheduled start time` from the task at all

Jeremiah · Answer

So I m still looking in to figure out why you re seeing that behavior

Jeremiah · Answer

though I think we actually should set a task scheduled start time for tasks that are in fact scheduled

Jeremiah · Answer

Ah I think I understand This is a limitation of how we run flows in process I ll need to talk to < Chris White> about the best way to solve it Essentially we create a new `FlowRunner` class for each scheduled event In this case that means creating a new `FlowRunner` for the retrying task in one minute Because the in process runner doesn t have a database attached it doesn t know what flow run originated the retry in question so it just sets the `scheduled start time` to the current time

Jeremiah · Answer

There are a couple approaches we can take to persist the appropriate context but I want to talk to Chris before I volunteer a solution

Mikhail Akimov · Answer

Please let me know where I can track the status of this issue or just ping me here when you come to a solution

Mikhail Akimov · Answer

As a temporary workaround one can make the first task of the flow return it s scheduled start time and then use it as an input for subsequent tasks so that it s captured for possible retries

Jeremiah · Answer

Based on my first impression I ve opened the following PR to more clearly name the flow run context variable <https github com PrefectHQ prefect pull 1403> Will follow up on the other issue

Jeremiah · Answer

That should work for the moment but hopefully we can come up with a first class solution

Mikhail Akimov · Answer

< Jeremiah> Hi Any news on this I saw the other pull requests but they were more about naming than this particular problem

Jeremiah · Answer

< Mikhail Akimov> my apologies for neglecting to update you I believe the solution is actually a lot simpler than I thought but I still need to write a couple tests to confirm

Jeremiah · Answer

If all goes well look for a PR today or tomorrow

Mikhail Akimov · Answer

This is great thanks

Jeremiah · Answer

< Mikhail Akimov> Just PRd which I hope solves the issue Fortunately the fix was quite simple hats off to < Chris White> for his good design

Jeremiah · Answer

< Mikhail Akimov> this PR was merged tonight so you can take advantage of this by installing Prefect `master` Otherwise it will be in the next release that we expect to cut at the end of this week

Mikhail Akimov · Answer

Thanks a lot for your help Jeremiah

Mikhail Akimov · Answer

Hi < Jeremiah> Please take a look

Jeremiah · Answer

On it right now honestly not sure why offhand but will have a fix shortly

Jeremiah · Answer

Believe this will do it <https github com PrefectHQ prefect pull 1429>

Mikhail Akimov · Answer

Thanks