https://prefect.io
Join Slack
HI, I am noticing a weird bug. I see many of my flow runs in PENDING state. I have kubernetes infra....
# ask-community
d
HI, I am noticing a weird bug. I see many of my flow runs in PENDING state. I have kubernetes infra. While looking at the pod list it seems that the pods associated with the Flow run are in ERROR state. The logs show "Temporary failure in name resolution" error message. Why are the flow runs in PENDING state and not CRASHED ?
Code flow has not yet entered the flow method yet. The error is originating on the import block of the file containing the flow.
Copy code
import stuff #error here

@flow
def ......:
    ....
j
Are you using a Kubernetes infrastructure block or a Kubernetes Worker?
d
infrastucture block
the kubernetes agent is deployed using helm chart provided by prefect
j
To get a crashed flow, the agent/worker has to be able to get a response back from the state of the infra. It's possible the 
Temporary failure in name resolution
is stopping the status from coming back?
d
And when does the first report of status occur?
The run has been submitted to run, but is waiting on necessary preconditions to be satisfied.
According to docs. not sure what the precoditions are
j
After the job is created, it's checked for completion via 
watch.stream()
do you have agent logs? it might help
d
The temp error in name resolution is not prefect issue.
basically my code has a global variable , value of which is fetched from Azure key vault. Thats where the error is occuring.
I would still except the status on prefect to be Failed OR Crashed.
its been an hour šŸ˜„
watch.stream()
is this an interval polling system or event based trigger system?
j
What's your container's exit code?
d
let me try describe the pod
j
What I think is happening is: ā€¢ your pod is crashing from the import-ish error ā€¢ the agent can't get the error code back from the infra due to the k8s networking error ā€¢ the agent will only crash your flow if it gets a non 0 status code back from k8s you may see a log message like
Copy code
f"An error occured while monitoring flow run '{flow_run.id}'. The flow run will not be marked as failed, but an issue may have occurred."
d
Exit code is 1
Copy code
State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 20 Jul 2023 17:23:36 +0000
      Finished:     Thu, 20 Jul 2023 17:24:35 +0000
    Ready:          False
j
gotcha
if it was getting back to the agent you're correct it would report crashed
d
Is this expected tho?
or even intended
j
I'd look for the log message I sent above to confirm
but it is expected that if there is an unexpected exception while monitoring the flow run would not crash
for example, if you fixed the import error
d
Is that error mesage on the agent or the server?
j
the agent would not want to crash your flow, just cause it couldnt reach k8s
it would be in the agent logs
d
Well thats not goign to be easy to find cuz I have 10 agents running. šŸ˜„
on different machines
j
šŸ˜…
if it was a worker, you would be able to see it in the prefect logs for that flow run I believe
d
Copy code
try:
	import stuff
except Exception as err:
	fail_import=True
else:
	fail_import=False

@flow
def a():
	if fail_import:
		raise RuntimeError("Import failed {err}")
šŸ‘€
its an agent not a worker. we havnt movedto workers yet because there are some issues regarding permissions on user scope on helm deployments.
j
gotcha make sense
d
But wouldnt itmake sense to flag a flow run as failed after X numer of attempts ?
I see around 5 pods were created so it did make 5 attempts and just gave up after that.
I manually cancelled the flow and started it. Now its complete.
This is painful as this blocks workqueues with limited concurrency.
technically stopping the pipeline
Or does Pending flows not take a slot on work queues?
j
If everything is working as intended (without the network issue i'm guessing) the status code would be reported as 1 and the flow run would crash
I believe 
PENDING
, 
RUNNING
and 
CANCELLING
count as in progress towards a work queue
while monitoring the agent functions on the policy that if can't reach the container
it's not enough to crash the flow. if the import error didn't exist for example, the flow run would finish fine, even though the agent could not monitor it, if that makes sense
I don't believe you're using Cloud if I remember right, but that is certainly a case an automation could handle (in 
PENDING
for X minutes)
d
Id except any exception raised by the script I wrote should have been marked as failed.
j
do you have logs from the flow run container? depending on the error maybe it should be crashed? it depends on when/how it's happening
d
The prefect server and the prefect agents are running in the same machines have connection error as well. Like I said its not termination of connection between the server and the agent but rather the connection to azure services.
Do u mean the logs from the pod?
j
yes
d
yes i do
OOf it just got cleared
j
ahhh darn.
depending on what's happening on the container it could report your flow as failed/crashed directly, but it's hard to tell without logs
I guess I was assuming the container wasn't able to even begin the entrypoint
d
Found 1
Copy code
kubectl logs  piquant-teal-jh567-dlpgh -n prefect2-worker-prod
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/app/my_library/__init__.py", line 5, in <module>
      ======CODE REDACTED=====
      but my_library/__init__.py import is callint azure core functions below




  File "/app/.venv/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 76, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/app/.venv/lib/python3.10/site-packages/azure/keyvault/secrets/_client.py", line 72, in get_secret
    bundle = self._client.get_secret(
  File "/app/.venv/lib/python3.10/site-packages/azure/keyvault/secrets/_generated/_operations_mixin.py", line 1640, in get_secret
    return mixin_instance.get_secret(vault_base_url, secret_name, secret_version, **kwargs)
  File "/app/.venv/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 76, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/app/.venv/lib/python3.10/site-packages/azure/keyvault/secrets/_generated/v7_4/operations/_key_vault_client_operations.py", line 760, in get_secret
    pipeline_response: PipelineResponse = self._client._pipeline.run(  # pylint: disable=protected-access
  File "/app/.venv/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 202, in run
    return first_node.send(pipeline_request)
  File "/app/.venv/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 70, in send
    response = self.next.send(request)
  File "/app/.venv/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 70, in send
    response = self.next.send(request)
  File "/app/.venv/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 70, in send
    response = self.next.send(request)
  [Previous line repeated 2 more times]
  File "/app/.venv/lib/python3.10/site-packages/azure/core/pipeline/policies/_redirect.py", line 156, in send
    response = self.next.send(request)
  File "/app/.venv/lib/python3.10/site-packages/azure/core/pipeline/policies/_retry.py", line 470, in send
    raise err
  File "/app/.venv/lib/python3.10/site-packages/azure/core/pipeline/policies/_retry.py", line 448, in send
    response = self.next.send(request)
  File "/app/.venv/lib/python3.10/site-packages/azure/core/pipeline/policies/_authentication.py", line 113, in send
    response = self.next.send(request)
  File "/app/.venv/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 70, in send
    response = self.next.send(request)
  File "/app/.venv/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 70, in send
    response = self.next.send(request)
  File "/app/.venv/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 70, in send
    response = self.next.send(request)
  [Previous line repeated 1 more time]
  File "/app/.venv/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 101, in send
    self._sender.send(request.http_request, **request.context.options),
  File "/app/.venv/lib/python3.10/site-packages/azure/core/pipeline/transport/_requests_basic.py", line 364, in send
    raise error
azure.core.exceptions.ServiceRequestError: <urllib3.connection.HTTPSConnection object at 0x7f56dc3fc670>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution
j
what storage are you using? is the flow code just in the image or it's remote?
d
its in an image
we basically build an image push it to ACR and create a prefect deployment pointing to thatimage
j
is there a traceback around that that is from prefect code? or any prefect logs? if not I don't think the python process is getting off the ground. So the prefect process on that container never starts and doesn't have an opportunity to crash the flow the agent would crash this if the error code made it's way fully back
how consistent are those DNS errors? i'd expect that to only happen some of the time? and other times it should crash ?
d
Thats the entire log,
This is the first time in 6 months that ive seen the DNS error
j
if you're able to see this happen consistently with a reproducible example and record the agent logs it would be great if you could file an issue for it!
d
I think that should be easy. Ill get to it when i can.
šŸ™Œ 1
Thanks for the help.
4 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by