https://prefect.io logo
Powered by
Title
a

Alexander van Eck

08/04/2021, 1:41 PM
Morning 👋 I’m using 
DaskExecutor
with a custom 
cluster_class
the logs in Prefect UI show me; 
15:36:12 INFO DaskExecutor Creating a new Dask cluster with `dask_jobqueue.remote_slurm.RemoteSLURMCluster.with_http_api_client`...
15:36:13 INFO DaskExecutor The Dask dashboard is available at <http://10.240.3.45:8787/status>
But I know for a fact that RemoteSlurmCluster (by dask-jobqueue) also has logger.debug statement. How do I make those logger statements show up in either the flow logs (in the agent) or in the prefect logger UI?
k

Kevin Kho

08/04/2021, 1:47 PM
Hey @Alexander van Eck , morning! I think you’re trying to add extra loggers like this?
a

Alexander van Eck

08/04/2021, 1:49 PM
Interesting!
I suppose 
export PREFECT__LOGGING__EXTRA_LOGGERS="['']"
would log everything?
k

Kevin Kho

08/04/2021, 1:55 PM
Yeah you can add it to the 
RunConfig
. You just need to add the name of the module like 
dask-jobquue
. Also, just to be sure, you need to lower your logging level to debug with 
PREFECT___LOGGING___LEVEL="DEBUG"
a

Alexander van Eck

08/04/2021, 1:55 PM
Thanks 🙂 I’ll test this out
hm 🤔 @Kevin Kho I did 
'PREFECT__LOGGING__EXTRA_LOGGERS': "['dask_jobqueue']",
'PREFECT__LOGGING__LEVEL': 'DEBUG',
in RunConfig but that doesn’t show the logging of dask_jobqueue in Prefect UI or Prefect DockerAgent
k

Kevin Kho

08/04/2021, 3:10 PM
This the debug level logs at least work?
a

Alexander van Eck

08/04/2021, 3:25 PM
I can see prefect debug logs yes (
Checking flow run state...
mostly) but not anything from 
dask_jobqueue
k

Kevin Kho

08/04/2021, 3:33 PM
Maybe we got the name of the logger wrong? Would you know the logger name?
a

Alexander van Eck

08/04/2021, 3:55 PM
I don’t think we did… 😬_ it’s being instantiated with It gets it’s name from 
__name__
k

Kevin Kho

08/04/2021, 3:56 PM
Yeah that would be the module name. Do you see the logs when you run on local with 
flow.run()
?
a

Alexander van Eck

08/04/2021, 3:59 PM
The DockerAgent I’m also running is showing them yes
Aug  4 16:00:50 workflow-01 prefect[10697]: [2021-08-04 16:00:41+0000] DEBUG - dask_jobqueue.core | Starting job: 54253
I’m not sure why --show-flow-logs in the agent and the Prefect UI wouldn’t show the same logs given that the level is debug in both.
k

Kevin Kho

08/04/2021, 4:09 PM
If you didn’t have the env variable, it wouldn’t even show up on the agent right?
a

Alexander van Eck

08/04/2021, 4:31 PM
Correct - I think it might be because our server is 
0.14.21
and the agent is 
0.15.3
do you know?
k

Kevin Kho

08/04/2021, 4:35 PM
I don’t have ideas yet, but I don’t think it’s that. What is your executor?
a

Alexander van Eck

08/04/2021, 4:51 PM
DaskExecutor
k

Kevin Kho

08/04/2021, 4:54 PM
Oh yeah of course 🤦‍♂️. Let me ask the team for any ideas and I’ll get back to you when there are.
Wait sorry just wanna be sure I have all the details right. You see the debug level logs on the agent correctly, but not on the UI, but you see other debug level logs on the UI right?
a

Alexander van Eck

08/05/2021, 1:35 PM
@Kevin Kho Correct - I see the debug logs I would expect in the Agent logs but not in the UI. I see only prefect debug messages, specifically only 
Checking flow run state...
.
k

Kevin Kho

08/05/2021, 2:44 PM
Unfortunately, I have no news for you yet. Will follow up with the team.
a

Alexander van Eck

08/17/2021, 3:23 PM
@Kevin Kho 👋 How’ve you been? 🙂 Any word on this issue?
k

Kevin Kho

08/17/2021, 3:26 PM
Hey, I have no ideas on this issue.
The only thing I can possibly think of is that you might have the UI set to 
INFO
logs? Do you see the filter in the UI? If it’s displaying on the agent with --show-flow-logs, it should be in the UI. Can you show me what the agent logs look like? I wanna see if it’s formatted like the Prefect logger.
Went through this again. Looks like you have the UI not filtering logs because you saw the DEBUG logs. Could you try a LocalDaskExecutor perhaps and see if that makes it to the UI?
Actually I think I have time today let me run a test.
a

Alexander van Eck

08/17/2021, 3:39 PM
Thank you 🙂 Unfortunately we are in different timezones and I’m just about to stop for the day. I’ll look into this tomorrow (Central European timezone)
I look forward to anything you find 🙌 and I’ll try to gather more info as per your comments
k

Kevin Kho

08/17/2021, 3:39 PM
Yeah I’ll give this a shot myself
So I talked to the team, and the likely reason for this behavior is that the 
dask-jobqueue
is running on a different process than the flow. because of that, the CloudHandler is not getting attached to the logger, so I’m not sure 
dask-jobqueue
logs can easily be brought to the Prefect UI
2 Views
#prefect-serverJoin Slack
Powered by