Hello. I have a flow thats permanently in a PENDIN...
# ask-communityd
Deceivious
05/31/2023, 8:42 AMHello. I have a flow thats permanently in a PENDING state? How do I begin to debug this issue? Other flows in the same work queue is RUNNING and completing successfully.
✅ 1
j
Jake Kaplan
05/31/2023, 11:36 AMHey, do you have any logs related to the flow run submission?
d
Deceivious
05/31/2023, 11:43 AM@Jake Kaplan where can I get it from?
j
Jake Kaplan
05/31/2023, 11:46 AMIt'd be from your agent/worker. If you're using a worker some of the logs from the submission should be attached to your flow run logs in the UI, otherwise anything from running your agent/worker
d
Deceivious
05/31/2023, 2:03 PM@Jake Kaplan Thanks for the guidance.
The issue seems to be on my side. I seem to have messed up the path to my config file.
So python native FileNotFoundError is being raised. But why is the flow run in Pending state. Should it not be in Failed state?
j
Jake Kaplan
05/31/2023, 2:06 PMare you able to see where that error is being raised from?
d
Deceivious
05/31/2023, 2:07 PMYes it is from within my code.
Deceivious
05/31/2023, 2:07 PMso a basic
with openj
Jake Kaplan
05/31/2023, 2:07 PMdo you have a stacktrace or even better flowrun logs?
d
Deceivious
05/31/2023, 2:08 PMPosting it
Deceivious
05/31/2023, 2:08 PM Copy code
$kubectl logs deft-saluki-xj58k-v4zdm -n <namespacehere>
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/app/generic_folder/__init__.py", line 5, in <module>
from generic_folder.prefect_structs.extended_tasks import (
File "/app/generic_folder/prefect_structs/extended_tasks.py", line 5, in <module>
from generic_folder.utils.prefect_utils import (
File "/app/generic_folder/utils/prefect_utils.py", line 18, in <module>
from generic_folder.config import IS_LOCAL_DEV, config
File "/app/generic_folder/config.py", line 9, in <module>
config = KeyvaultConfigManager(path=f"{os.getenv('ENVIRONMENT')}.toml")
File "/app/.venv/lib/python3.10/site-packages/archimedes_config/keyvault_config_manager.py", line 28, in __init__
super().__init__(
File "/app/.venv/lib/python3.10/site-packages/archimedes_config/config_manager.py", line 35, in __init__
self.load(path=path)
File "/app/.venv/lib/python3.10/site-packages/archimedes_config/config_manager.py", line 216, in load
with open(path, "r", encoding="utf8") as file:
FileNotFoundError: [Errno 2] No such file or directory: 'dev.toml'Deceivious
05/31/2023, 2:09 PMDeceivious
05/31/2023, 2:09 PMHeres what UI looks like
j
Jake Kaplan
05/31/2023, 2:09 PMand the logs tab?
d
Deceivious
05/31/2023, 2:10 PMThis run didn't generate LogsDeceivious
05/31/2023, 2:10 PMI am using an agent
j
Jake Kaplan
05/31/2023, 2:11 PMWhat you're seeing might be a different error or from a different run, since it looks like it never made it to a running state
Jake Kaplan
05/31/2023, 2:15 PMdo you have any logs from that flow from the agent?
d
Deceivious
05/31/2023, 2:17 PMI am pretty sure that the pod is associated with the flow because the pod creation time is after the scheduled time
j
Jake Kaplan
05/31/2023, 2:18 PMdo you have the rest of the logs from that pod?
d
Deceivious
05/31/2023, 2:18 PMThats the entire log
j
Jake Kaplan
05/31/2023, 2:21 PMoh hm it looks like it's erroring on import?
d
Deceivious
05/31/2023, 2:22 PMNope. Its an error on Class initialization.
Deceivious
05/31/2023, 2:22 PM Copy code
KeyvaultConfigManagerDeceivious
05/31/2023, 2:23 PMEither way, i think prefect should be classify this as an Error state and not Pending. DO you know if adding an ENTRYPOINT on the dockerfile for image has conflicts with prefect flow deployment?
j
Jake Kaplan
05/31/2023, 2:24 PMI think you might be overriding our default entrypoint 😅
d
Deceivious
05/31/2023, 2:25 PMYes i think so too.
j
Jake Kaplan
05/31/2023, 2:25 PMwe can't mark the flow as failed cause prefect is effectively not running
d
Deceivious
05/31/2023, 2:25 PMYes this kind of makes sense to me now
Deceivious
05/31/2023, 2:26 PMIll test after removing the entrypoint I added to see if i get a failure
👍 1
Deceivious
05/31/2023, 2:34 PMSo seems like the ENTRYPOINT does matter I commented the entrypoint code and it passed successfully.
Deceivious
05/31/2023, 2:38 PMWhat is the default entrypoint for prefect?
j
Jake Kaplan
05/31/2023, 2:42 PMJake Kaplan
05/31/2023, 2:42 PMI think the important part is that it's called with the right command
Jake Kaplan
05/31/2023, 2:43 PM Copy code
python -m prefect.engined
Deceivious
05/31/2023, 5:34 PMI am still using the endpoint and just changed the work directory and the issue fixed itself.
Deceivious
05/31/2023, 5:34 PMThanks!
15 Views