https://prefect.io logo
Join Slack
Powered by
Hi! When flow/task is retrying, in, say, 1 minute,...
# ask-community
m
Hi! When flow/task is retrying, in, say, 1 minute, it's scheduled_start_time in context is +1 minute from the failed one. I'm using scheduled_start_time kind of how I would use DagRun's timestamp key in Airflow (in an SQL template) so this behaviour isn't helpful. Am I doing it wrong? Can you get the original task's metadata/context?
j
Hi @Mikhail Akimov! Good question. To be clear, you want the start time (or scheduled start time) of the overall flow run itself, not the task run, right?
I’m looking in the code and realizing that we store both under the same context key 
scheduled_start_time
, so I believe the task value overwrites the flow value
m
Oh, so the original flow start time kind of should be there, but overwritten?
j
Yep, and I’m going to PR a fix right now
m
Super nice, thank you!
j
Thank you for pointing this out!
Actually I apologize, we don’t set the 
scheduled_start_time
from the task at all
So I’m still looking in to figure out why you’re seeing that behavior
(though I think we actually should set a task scheduled start time for tasks that are, in fact, scheduled!)
Ah, I think I understand. This is a limitation of how we run flows in-process. I’ll need to talk to @Chris White about the best way to solve it. Essentially: we create a new 
FlowRunner
class for each scheduled event. In this case, that means creating a new 
FlowRunner
for the retrying task in one minute. Because the in-process runner doesn’t have a database attached, it doesn’t know what flow run originated the retry in question, so it just sets the 
scheduled_start_time
to the current time
There are a couple approaches we can take to persist the appropriate context, but I want to talk to Chris before I volunteer a solution
m
Please let me know where I can track the status of this issue, or just ping me here when you come to a solution.
👍 1
As a temporary workaround, one can make the first task of the flow return it's scheduled_start_time and then use it as an input for subsequent tasks, so that it's captured for possible retries.
j
Based on my first impression, I’ve opened the following PR to more clearly name the flow run context variable: https://github.com/PrefectHQ/prefect/pull/1403. Will follow up on the other issue!
That should work for the moment, but hopefully we can come up with a first-class solution
m
@Jeremiah Hi! Any news on this? I saw the other pull requests, but they were more about naming than this particular problem.
j
@Mikhail Akimov, my apologies for neglecting to update you — I believe the solution is actually a lot simpler than I thought, but I still need to write a couple tests to confirm
If all goes well, look for a PR today or tomorrow
m
This is great, thanks!
j
@Mikhail Akimov Just PRd https://github.com/PrefectHQ/prefect/pull/1418 which I hope solves the issue. Fortunately the fix was quite simple — hats off to @Chris White for his good design
👍 1
😎 1
@Mikhail Akimov this PR was merged tonight, so you can take advantage of this by installing Prefect 
master
. Otherwise, it will be in the next release that we expect to cut at the end of this week.
m
Thanks a lot for your help Jeremiah!
j
On it right now, honestly not sure why offhand but will have a fix shortly
m
Thanks!
Open in Slack
PreviousNext
Powered by