https://prefect.io logo
#prefect-community
Title
# prefect-community
m

Mikhail Akimov

08/24/2019, 11:26 PM
Hi! When flow/task is retrying, in, say, 1 minute, it's scheduled_start_time in context is +1 minute from the failed one. I'm using scheduled_start_time kind of how I would use DagRun's timestamp key in Airflow (in an SQL template) so this behaviour isn't helpful. Am I doing it wrong? Can you get the original task's metadata/context?
j

Jeremiah

08/24/2019, 11:30 PM
Hi @Mikhail Akimov! Good question. To be clear, you want the start time (or scheduled start time) of the overall flow run itself, not the task run, right?
I’m looking in the code and realizing that we store both under the same context key
scheduled_start_time
, so I believe the task value overwrites the flow value
m

Mikhail Akimov

08/24/2019, 11:31 PM
Oh, so the original flow start time kind of should be there, but overwritten?
j

Jeremiah

08/24/2019, 11:31 PM
Yep, and I’m going to PR a fix right now
m

Mikhail Akimov

08/24/2019, 11:31 PM
Super nice, thank you!
j

Jeremiah

08/24/2019, 11:31 PM
Thank you for pointing this out!
Actually I apologize, we don’t set the
scheduled_start_time
from the task at all
So I’m still looking in to figure out why you’re seeing that behavior
(though I think we actually should set a task scheduled start time for tasks that are, in fact, scheduled!)
Ah, I think I understand. This is a limitation of how we run flows in-process. I’ll need to talk to @Chris White about the best way to solve it. Essentially: we create a new
FlowRunner
class for each scheduled event. In this case, that means creating a new
FlowRunner
for the retrying task in one minute. Because the in-process runner doesn’t have a database attached, it doesn’t know what flow run originated the retry in question, so it just sets the
scheduled_start_time
to the current time
There are a couple approaches we can take to persist the appropriate context, but I want to talk to Chris before I volunteer a solution
m

Mikhail Akimov

08/25/2019, 12:00 AM
Please let me know where I can track the status of this issue, or just ping me here when you come to a solution.
👍 1
As a temporary workaround, one can make the first task of the flow return it's scheduled_start_time and then use it as an input for subsequent tasks, so that it's captured for possible retries.
j

Jeremiah

08/25/2019, 12:08 AM
Based on my first impression, I’ve opened the following PR to more clearly name the flow run context variable: https://github.com/PrefectHQ/prefect/pull/1403. Will follow up on the other issue!
That should work for the moment, but hopefully we can come up with a first-class solution
m

Mikhail Akimov

08/28/2019, 12:06 PM
@Jeremiah Hi! Any news on this? I saw the other pull requests, but they were more about naming than this particular problem.
j

Jeremiah

08/28/2019, 12:12 PM
@Mikhail Akimov, my apologies for neglecting to update you — I believe the solution is actually a lot simpler than I thought, but I still need to write a couple tests to confirm
If all goes well, look for a PR today or tomorrow
m

Mikhail Akimov

08/28/2019, 12:20 PM
This is great, thanks!
j

Jeremiah

08/28/2019, 3:01 PM
@Mikhail Akimov Just PRd https://github.com/PrefectHQ/prefect/pull/1418 which I hope solves the issue. Fortunately the fix was quite simple — hats off to @Chris White for his good design
👍 1
😎 1
@Mikhail Akimov this PR was merged tonight, so you can take advantage of this by installing Prefect
master
. Otherwise, it will be in the next release that we expect to cut at the end of this week.
m

Mikhail Akimov

08/29/2019, 8:55 AM
Thanks a lot for your help Jeremiah!
j

Jeremiah

08/30/2019, 11:54 AM
On it right now, honestly not sure why offhand but will have a fix shortly
m

Mikhail Akimov

08/30/2019, 12:43 PM
Thanks!