at the moment and have tried setting it as an environment variable for the agent, as well as adding it to the config.toml on the machine hosting prefect server but I'm still seeing
ValueError: Local Secret "SLACK_WEBHOOK_URL" was not found.
and then it never actually retries.. I believe the Lazarus process must kick in every 10 minutes and reschedule the task, right? CC: @Christina Lopez @Kevin Kho @Anna Geller
No heartbeat detected from the remote task; retrying the run.This will be retry 1 of 3.
Sometimes the flow can continue without any problem, because we have infinite flow set to continue even when flow failed:
Failed to retrieve task state with error: ReadTimeout(ReadTimeoutError("HTTPConnectionPool(host='apollo', port=4200): Read timed out. (read timeout=15)"))
def never_ending_state_handler(obj, old_state, new_state):
if (old_state.is_running() and new_state.is_failed()):
if (old_state.is_running() and new_state.is_successful()) or (old_state.is_running() and new_state.is_failed()):
create_flow_run.run(flow_name="our_flow", project_name = "our_project", run_name = str(uuid.uuid4()))
But when we receive error:
we are not able to continue... The flow is failed but state handler is not working anymore to reschedule failed flow. Does anybody know what may cause problem with Read timed out or Lazarus?
A Lazarus process attempted to reschedule this run 3 times without success. Marking as failed.
within a task created by a
... assuming i have
with Flow("flow-1", run_config=KubernetesRun(...)): ... creates task that does a create_flow_run.run(flow_name="flow-2") ...
Timeout waiting for network interface provisioning to complete.
. I have created a subclass of ShellTask to invoke the spark-submit.The spark jobs are running on k8s. I am facing an issue where prefect tasks are not completing and continuously running. I see this happening on the tasks which are slightly long running (> 10 mins). The master flow maps over list and orchestrates the prefect
The task starts running for
K8sSparkSubmitTask.map( id = ["id1", "id2", "idx"] )
and pod status is observed to be
. However prefect does not move to the next task. Why are these tasks not getting completed?
results in an error :
helm pull --destination /tmp/c68c98cd-c919-46b6-9e17-924f22aa2dd3 --version 2022.01.25 --repo <https://prefecthq.github.io/server> prefecthq/prefect-server
However, I do see
Error: chart "prefecthq/prefect-server" version "2022.01.25" not found in <https://prefecthq.github.io/server> repository
as a version when I do
. Can someone please point out where I am going wrong?
helm search repo prefecthq/prefect-server --versions