Tony Yun
10/22/2021, 8:01 PMKevin Kho
raise ENDRUN
. This will avoid any retries. Docs . Is this what you meant?Tony Yun
10/22/2021, 8:05 PMTony Yun
10/22/2021, 8:08 PMwith xxx as flow:
a = taskA()
# how to kill task a as a task?
Kevin Kho
RunNamespacedJob
that spins up new compute like this:
@task
def task_a():
RunNamespacedJob().run()
return
In this case, RunNamespacedJob
is not a real task and in general, Prefect is currently not able to close hardware through cancelletationTony Yun
10/22/2021, 8:10 PMKevin Kho
task_a
mid flow run through the flow is very hard. Because it must have already executed by the time downstream tasks are reachedTony Yun
10/22/2021, 8:13 PMRunNamespacedJob().run()
like task_id, I can call something like client.set_task_run_state()
Kevin Kho
RunNamespacedJob
from a downstream task unless you persist it in some location that can be retrieved from downstream.
If you are getting the metadata in the same task,you would need a modified task that allows you to poll the NamespacedJob or something or check the logs, and then you can have a conditional that raise SUCCESS
or raise FAIL
. These would be better than the client.set_task_run_state()
if the logic is in the same task.Tony Yun
10/23/2021, 12:26 AM