https://prefect.io logo
Join Slack
Powered by
Hi everyone — I have a flow that runs correctly wi...
# ask-community
b
Hi everyone — I have a flow that runs correctly with “prefect run”, however if I try to run via local agent then I get a weird error and the flow hangs in the “submitted” state
k
Hey @Bruno Murino, are you using the --watch flag? Could you try running debug logs on the agent?
b
@Kevin Kho I’m deploying the agent via Python API with
Copy code
LocalAgent(
        name=AGENT_NAME,
        agent_address="<http://0.0.0.0:8009>",
    ).start()
I’m trying to find an option to use to have debug logs but can’t find
s
The log level is configurable with the environment variable PREFECT__LOGGING__LEVEL https://docs.prefect.io/core/concepts/logging.html#logging-configuration
b
doesn’t look like this option is being applied to agents
k
I think you might have to do it through the config.toml like
Copy code
# ~/.prefect/config.toml
[cloud.agent]
level = "debug"
Will test now
b
for now I just started the agent from the CLI to see the logs
k
level = “DEBUG” will work
👍 1
b
Copy code
[2021-07-26 15:09:28,853] DEBUG - BM Health Monitoring | Querying for ready flow runs...
[2021-07-26 15:09:29,035] DEBUG - BM Health Monitoring | No ready flow runs found.
[2021-07-26 15:09:29,036] DEBUG - BM Health Monitoring | Sleeping flow run poller for 10.0 seconds...
[2021-07-26 15:09:39,038] DEBUG - BM Health Monitoring | Querying for ready flow runs...
[2021-07-26 15:09:39,207] DEBUG - BM Health Monitoring | Found 1 ready flow run(s): {'1b5112ae-00ba-4e35-9d98-9d61a3d82bf5'}
[2021-07-26 15:09:39,208] DEBUG - BM Health Monitoring | Retrieving metadata for 1 flow run(s)...
[2021-07-26 15:09:39,359] DEBUG - BM Health Monitoring | Submitting flow run 1b5112ae-00ba-4e35-9d98-9d61a3d82bf5 for deployment...
[2021-07-26 15:09:39,364] INFO - BM Health Monitoring | Deploying flow run 1b5112ae-00ba-4e35-9d98-9d61a3d82bf5 to execution environment...
[2021-07-26 15:09:39,364] DEBUG - BM Health Monitoring | Sleeping flow run poller for 0.25 seconds...
[2021-07-26 15:09:39,365] DEBUG - BM Health Monitoring | Updating flow run 1b5112ae-00ba-4e35-9d98-9d61a3d82bf5 state from Scheduled -> Submitted...
[2021-07-26 15:09:39,546] DEBUG - BM Health Monitoring | Submitted flow run 1b5112ae-00ba-4e35-9d98-9d61a3d82bf5 to process PID 2093
[2021-07-26 15:09:39,546] INFO - BM Health Monitoring | Completed deployment of flow run 1b5112ae-00ba-4e35-9d98-9d61a3d82bf5
[2021-07-26 15:09:39,615] DEBUG - BM Health Monitoring | Querying for ready flow runs...
[2021-07-26 15:09:39,772] DEBUG - BM Health Monitoring | No ready flow runs found.
[2021-07-26 15:09:39,772] DEBUG - BM Health Monitoring | Sleeping flow run poller for 0.5 seconds...
[2021-07-26 15:09:40,274] DEBUG - BM Health Monitoring | Querying for ready flow runs...
[2021-07-26 15:09:40,434] DEBUG - BM Health Monitoring | No ready flow runs found.
[2021-07-26 15:09:40,434] DEBUG - BM Health Monitoring | Sleeping flow run poller for 1.0 seconds...
[2021-07-26 15:09:41,437] DEBUG - BM Health Monitoring | Querying for ready flow runs...
[2021-07-26 15:09:41,618] DEBUG - BM Health Monitoring | No ready flow runs found.
[2021-07-26 15:09:41,619] DEBUG - BM Health Monitoring | Sleeping flow run poller for 2.0 seconds...
[2021-07-26 15:09:42,027] DEBUG - BM Health Monitoring | Sending agent heartbeat...
[2021-07-26 15:09:42,027] INFO - BM Health Monitoring | Process PID 2093 returned non-zero exit code 1!
[2021-07-26 15:09:42,027] DEBUG - BM Health Monitoring | Heartbeat succesful! Sleeping for 60.0 seconds...
[2021-07-26 15:09:43,621] DEBUG - BM Health Monitoring | Querying for ready flow runs...
[2021-07-26 15:09:43,795] DEBUG - BM Health Monitoring | No ready flow runs found.
[2021-07-26 15:09:43,796] DEBUG - BM Health Monitoring | Sleeping flow run poller for 4.0 seconds...
[2021-07-26 15:09:47,796] DEBUG - BM Health Monitoring | Querying for ready flow runs...
[2021-07-26 15:09:47,950] DEBUG - BM Health Monitoring | No ready flow runs found.
[2021-07-26 15:09:47,950] DEBUG - BM Health Monitoring | Sleeping flow run poller for 8.0 seconds...
[2021-07-26 15:09:55,951] DEBUG - BM Health Monitoring | Querying for ready flow runs...
[2021-07-26 15:09:56,094] DEBUG - BM Health Monitoring | No ready flow runs found.
[2021-07-26 15:09:56,095] DEBUG - BM Health Monitoring | Sleeping flow run poller for 10.0 seconds...
[2021-07-26 15:10:06,095] DEBUG - BM Health Monitoring | Querying for ready flow runs...
[2021-07-26 15:10:06,279] DEBUG - BM Health Monitoring | Found 1 ready flow run(s): {'8651c9e3-817e-4ef9-87fd-2cc402ef1dc6'}
[2021-07-26 15:10:06,280] DEBUG - BM Health Monitoring | Retrieving metadata for 1 flow run(s)...
[2021-07-26 15:10:06,400] DEBUG - BM Health Monitoring | Submitting flow run 8651c9e3-817e-4ef9-87fd-2cc402ef1dc6 for deployment...
[2021-07-26 15:10:06,401] DEBUG - BM Health Monitoring | Sleeping flow run poller for 0.25 seconds...
[2021-07-26 15:10:06,403] INFO - BM Health Monitoring | Deploying flow run 8651c9e3-817e-4ef9-87fd-2cc402ef1dc6 to execution environment...
[2021-07-26 15:10:06,404] DEBUG - BM Health Monitoring | Updating flow run 8651c9e3-817e-4ef9-87fd-2cc402ef1dc6 state from Scheduled -> Submitted...
[2021-07-26 15:10:06,554] DEBUG - BM Health Monitoring | Submitted flow run 8651c9e3-817e-4ef9-87fd-2cc402ef1dc6 to process PID 2094
[2021-07-26 15:10:06,554] INFO - BM Health Monitoring | Completed deployment of flow run 8651c9e3-817e-4ef9-87fd-2cc402ef1dc6
[2021-07-26 15:10:06,657] DEBUG - BM Health Monitoring | Querying for ready flow runs...
[2021-07-26 15:10:06,782] DEBUG - BM Health Monitoring | No ready flow runs found.
[2021-07-26 15:10:06,783] DEBUG - BM Health Monitoring | Sleeping flow run poller for 0.5 seconds...
[2021-07-26 15:10:07,287] DEBUG - BM Health Monitoring | Querying for ready flow runs...
[2021-07-26 15:10:07,452] DEBUG - BM Health Monitoring | No ready flow runs found.
[2021-07-26 15:10:07,452] DEBUG - BM Health Monitoring | Sleeping flow run poller for 1.0 seconds...
[2021-07-26 15:10:08,453] DEBUG - BM Health Monitoring | Querying for ready flow runs...
[2021-07-26 15:10:08,583] DEBUG - BM Health Monitoring | No ready flow runs found.
[2021-07-26 15:10:08,584] DEBUG - BM Health Monitoring | Sleeping flow run poller for 2.0 seconds...
[2021-07-26 15:10:10,584] DEBUG - BM Health Monitoring | Querying for ready flow runs...
[2021-07-26 15:10:10,757] DEBUG - BM Health Monitoring | No ready flow runs found.
[2021-07-26 15:10:10,757] DEBUG - BM Health Monitoring | Sleeping flow run poller for 4.0 seconds...
[2021-07-26 15:10:14,758] DEBUG - BM Health Monitoring | Querying for ready flow runs...
[2021-07-26 15:10:14,888] DEBUG - BM Health Monitoring | No ready flow runs found.
[2021-07-26 15:10:14,888] DEBUG - BM Health Monitoring | Sleeping flow run poller for 8.0 seconds...
[2021-07-26 15:10:22,890] DEBUG - BM Health Monitoring | Querying for ready flow runs...
[2021-07-26 15:10:23,066] DEBUG - BM Health Monitoring | No ready flow runs found.
[2021-07-26 15:10:23,066] DEBUG - BM Health Monitoring | Sleeping flow run poller for 10.0 seconds...
[2021-07-26 15:10:33,069] DEBUG - BM Health Monitoring | Querying for ready flow runs...
[2021-07-26 15:10:33,212] DEBUG - BM Health Monitoring | No ready flow runs found.
[2021-07-26 15:10:33,212] DEBUG - BM Health Monitoring | Sleeping flow run poller for 10.0 seconds...
[2021-07-26 15:10:42,012] DEBUG - BM Health Monitoring | Sending agent heartbeat...
[2021-07-26 15:10:42,012] INFO - BM Health Monitoring | Process PID 2094 returned non-zero exit code 1!
not very helpful logs
k
Does that kill the agent also?
b
no, agent still UP and returns same message when trying to run other flows
I mean, other flows from same project
prefect version 0.15.1
@Kevin Kho do you by any change know a workaround this problem? It’s impairing our production system quite heavily as our most crucial flows are deployed with a local agent
k
I am wondering if this related to the flow. Were these flows working before? Does a simple flow fail also?
Is this only when you use Prefect run? Or does it also happen when triggered by the UI?
b
issue only happens when running via Agent — “prefect run” works just fine
let me try a dummy flow and see
ok, a dummy flow ran fine
k
The logs in this Flow show it got deployed. Do you see any logs on the UI?
b
only odd thing I have on my flow is a function “dispatcher”:
@Kevin Kho yea it ran fine, the dummy flow
k
I mean the old one that exited with the exit code 1
b
aah:
k
I think we need DEBUG level logs on the flow as well to diagnose this. You can do this through the UI by clicking the “Run” and then configuring it with the DEBUG level logs
b
didn’t change anything
k
Can you try adding it to the Flow RunConfig then and then re-registering 
DockerRun(env={"PREFECT___LOGGING____LEVEL: "DEBUG"_}
and then rerunning?
b
it’s not a dockerRun, but a “localRun”, but I’ll try that
@Kevin Kho didn’t change anything.. no new logs
k
That’s weird. Do you not even see any DEBUG logs at the start? There should be some saying where it downloaded stuff from. It does seem like it’s something specific to the flow though. Is your flow spinning up new processes?
b
these are the whole DEBUG logs
k
These are for the agent, but I was trying to see the logs on the Flow side as well through the UI. If you go to that Flow run in the UI, I assume none of the tasks ran right? Also, was the simple Flow in the same project? (I don’t think the project should affect anything but just making sure)
Could you show me your Storage if you have any and your RunConfig?
b
Nothing on the flow runs, not even the Parameter tasks, that’s why there are no Flow logs to show whatsoever
this is the file where I configure the flow
I already tried with/without the localDaskExecutor but didn’t change anything
k
When you tried the simple Flow? Did you get it working with the 
working_dir
?
b
no, I didn’t do any setup — now I removed all config from the flow and it ran just fine
I’ll try to narrow down what is the problematic bit
@Kevin Kho all is pointing to a problem with the “working_dir” config as you mentioned
k
Maybe try supplying the path to the Flow for 
Local
storage? Not expecting this to work but worth a shot.
What Prefect version are you on also?
b
apparently my flow works without the “LocalRun” config at all so I’m going with that for now
I’m on 0.15.1
I’ll try specifying flow path on Local storage
k
Sounds good! No need to specify the Flow Path then. It will work as long as the Local Agent is running from the directory you want your flow to be running in
b
yep it’s working fine!
👍 1
thank you very much for all the help @Kevin Kho, though it would be good to know why specifying working dir was causing issues haha
k
Yeah i’ll see what I can find.
You gave an abspath right?
b
yep
and it was working before today, with no version changes etc
2 Views
Open in Slack
PreviousNext
Powered by