https://prefect.io
Join Slack
Morning :wave: I’m using `DaskExecutor` with a custom `cluster_class` the logs in Prefect UI show m...
# prefect-server
a
Morning 👋 I’m using 
DaskExecutor
with a custom 
cluster_class
the logs in Prefect UI show me;
Copy code
15:36:12 INFO DaskExecutor Creating a new Dask cluster with `dask_jobqueue.remote_slurm.RemoteSLURMCluster.with_http_api_client`...
15:36:13 INFO DaskExecutor The Dask dashboard is available at <http://10.240.3.45:8787/status>
But I know for a fact that RemoteSlurmCluster (by dask-jobqueue) also has logger.debug statement. How do I make those logger statements show up in either the flow logs (in the agent) or in the prefect logger UI?
k
Hey @Alexander van Eck , morning! I think you’re trying to add extra loggers like this?
a
Interesting!
I suppose
Copy code
export PREFECT__LOGGING__EXTRA_LOGGERS="['']"
would log everything?
k
Yeah you can add it to the 
RunConfig
. You just need to add the name of the module like 
dask-jobquue
. Also, just to be sure, you need to lower your logging level to debug with 
PREFECT___LOGGING___LEVEL="DEBUG"
a
Thanks 🙂 I’ll test this out
hm 🤔 @Kevin Kho I did
Copy code
'PREFECT__LOGGING__EXTRA_LOGGERS': "['dask_jobqueue']",
'PREFECT__LOGGING__LEVEL': 'DEBUG',
in RunConfig but that doesn’t show the logging of dask_jobqueue in Prefect UI or Prefect DockerAgent
k
This the debug level logs at least work?
a
I can see prefect debug logs yes (
Checking flow run state...
mostly) but not anything from 
dask_jobqueue
k
Maybe we got the name of the logger wrong? Would you know the logger name?
a
I don’t think we did… 😬_ it’s being instantiated with It gets it’s name from 
__name__
k
Yeah that would be the module name. Do you see the logs when you run on local with 
flow.run()
?
a
The DockerAgent I’m also running is showing them yes
Copy code
Aug  4 16:00:50 workflow-01 prefect[10697]: [2021-08-04 16:00:41+0000] DEBUG - dask_jobqueue.core | Starting job: 54253
I’m not sure why --show-flow-logs in the agent and the Prefect UI wouldn’t show the same logs given that the level is debug in both.
k
If you didn’t have the env variable, it wouldn’t even show up on the agent right?
a
Correct - I think it might be because our server is 
0.14.21
and the agent is 
0.15.3
do you know?
k
I don’t have ideas yet, but I don’t think it’s that. What is your executor?
a
DaskExecutor
k
Oh yeah of course 🤦‍♂️. Let me ask the team for any ideas and I’ll get back to you when there are.
Wait sorry just wanna be sure I have all the details right. You see the debug level logs on the agent correctly, but not on the UI, but you see other debug level logs on the UI right?
a
@Kevin Kho Correct - I see the debug logs I would expect in the Agent logs but not in the UI. I see only prefect debug messages, specifically only 
Checking flow run state...
.
k
Unfortunately, I have no news for you yet. Will follow up with the team.
a
@Kevin Kho 👋 How’ve you been? 🙂 Any word on this issue?
k
Hey, I have no ideas on this issue.
The only thing I can possibly think of is that you might have the UI set to 
INFO
logs? Do you see the filter in the UI? If it’s displaying on the agent with --show-flow-logs, it should be in the UI. Can you show me what the agent logs look like? I wanna see if it’s formatted like the Prefect logger.
Went through this again. Looks like you have the UI not filtering logs because you saw the DEBUG logs. Could you try a LocalDaskExecutor perhaps and see if that makes it to the UI?
Actually I think I have time today let me run a test.
a
Thank you 🙂 Unfortunately we are in different timezones and I’m just about to stop for the day. I’ll look into this tomorrow (Central European timezone)
I look forward to anything you find 🙌 and I’ll try to gather more info as per your comments
k
Yeah I’ll give this a shot myself
So I talked to the team, and the likely reason for this behavior is that the 
dask-jobqueue
is running on a different process than the flow. because of that, the CloudHandler is not getting attached to the logger, so I’m not sure 
dask-jobqueue
logs can easily be brought to the Prefect UI
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by