Ismail Cenik
09/27/2021, 9:32 PMKevin Kho
09/27/2021, 9:45 PMIsmail Cenik
09/27/2021, 10:04 PMKevin Kho
09/27/2021, 10:07 PM@task
def new_task(app_name):
id = startKDA.run(app_name)
waitKDA.run(id)
return
that way they fail together. You can then use the boto3
timeout or prefect timeout to do:
@task(timeout=60, retry_info_here)
def new_task(app_name):
id = startKDA.run(app_name)
waitKDA.run(id)
return
and it would retry this block right?client.graphql()
call to the set_task_run_states
mutation and then get the `startKDA`task task_run_id and then set it to Scheduled (or Failed) and then restart the flow run.
You can do this in the state handler of the waitKDA
task. If waitKDA
fails, use graphql to mark startKDA
as failed also.Ismail Cenik
09/27/2021, 10:15 PMKevin Kho
09/27/2021, 10:16 PMIsmail Cenik
09/27/2021, 10:18 PMKevin Kho
09/27/2021, 10:23 PMdef my_state_handler(task, old_state, new_state):
if new_state.is_failed():
# get task id
task_id = client.graphql()
task_id = task_id[find_the_results_here]
# using task id, call set task run states to mark parevious as failed
client.graphql("""mutation {set_task_run_state{flow_id}}...""")
return new_state
waitKDA
task. You need two calls and some filtering to get the specific task if of the upstream startKDA
Ismail Cenik
09/27/2021, 10:41 PM@task(timeout=60, retry_info_here)
def new_task(app_name):
id = startKDA.run(app_name)
waitKDA.run(id)
return
Let's assume that I will use the above solution. I have 2 different flows which include different KDAs. When a timeout occurs, one of my flows will continue with the next step, however, the other one will retry the task. How can I specify these actions in the task?
And one more question is that can I provide a dynamic timeout value to the task? I have different thresholds value for each KDA. Will there be a problem with flow registration or another issue?Kevin Kho
09/27/2021, 10:44 PMIsmail Cenik
09/27/2021, 11:21 PM@task(log_stdout=True, timeout=3600, max_retries=2, retry_delay=timedelta(seconds=60))
One more thing, when a timeout occurs, before retrying or going to the next step, I want to call boto3 Stop KDA (force = true), since it is stuck, otherwise, the KDA cannot stop. So is it possible to run some code before retry?@task(log_stdout=True, timeout=3600, max_retries=2, retry_delay=timedelta(seconds=60), state_handlers=[my_state_handler])
So I can handle it inside my_state_handler, right?Kevin Kho
09/28/2021, 12:41 AMIsmail Cenik
09/30/2021, 6:30 PM@task(log_stdout=True, max_retries=5, retry_delay=timedelta(seconds=60))
def waitPlanDataLoadingTask(app_name, logId, parentLogId, envName):
I use python timeout and assume that timeout occurs after 10 mins later
Then I need to fail the flow, so when I use "raise signals.FAIL("Fail the flow")", the flow recovers itself because of the retry mechanism. So how can I bypass the retry mechanism without removing it? Should I use graphQL to make the flow fail? Could you please send a code example for this case?Kevin Kho
09/30/2021, 6:34 PMraise signals.ENDRUN
fails it without respecting the retries. You need something like raise signals.ENDRUN(Failed())