Morning wave I m using `DaskExecutor` with a custom `cluster Prefect Community #prefect-server

Morning :wave: I’m using `DaskExecutor` with a cu...

Alexander van Eck

08/04/2021, 1:41 PM

Morning 👋 I’m using

DaskExecutor

with a custom

cluster_class

the logs in Prefect UI show me;

Copy code

15:36:12 INFO DaskExecutor Creating a new Dask cluster with `dask_jobqueue.remote_slurm.RemoteSLURMCluster.with_http_api_client`...
15:36:13 INFO DaskExecutor The Dask dashboard is available at <http://10.240.3.45:8787/status>

But I know for a fact that RemoteSlurmCluster (by dask-jobqueue) also has logger.debug statement. How do I make those logger statements show up in either the flow logs (in the agent) or in the prefect logger UI?

Kevin Kho

08/04/2021, 1:47 PM

Hey @Alexander van Eck , morning! I think you’re trying to add extra loggers like this?

Alexander van Eck

08/04/2021, 1:49 PM

Interesting!

Alexander van Eck

08/04/2021, 1:49 PM

I suppose

Copy code

export PREFECT__LOGGING__EXTRA_LOGGERS="['']"

would log everything?

Kevin Kho

08/04/2021, 1:55 PM

Yeah you can add it to the

RunConfig

. You just need to add the name of the module like

dask-jobquue

. Also, just to be sure, you need to lower your logging level to debug with

PREFECT___LOGGING___LEVEL="DEBUG"

Alexander van Eck

08/04/2021, 1:55 PM

Thanks 🙂 I’ll test this out

Alexander van Eck

08/04/2021, 3:09 PM

hm 🤔 @Kevin Kho I did

Copy code

'PREFECT__LOGGING__EXTRA_LOGGERS': "['dask_jobqueue']",
'PREFECT__LOGGING__LEVEL': 'DEBUG',

in RunConfig but that doesn’t show the logging of dask_jobqueue in Prefect UI or Prefect DockerAgent

Kevin Kho

08/04/2021, 3:10 PM

This the debug level logs at least work?

Alexander van Eck

08/04/2021, 3:25 PM

I can see prefect debug logs yes (

Checking flow run state...

mostly) but not anything from

dask_jobqueue

Kevin Kho

08/04/2021, 3:33 PM

Maybe we got the name of the logger wrong? Would you know the logger name?

Alexander van Eck

08/04/2021, 3:55 PM

I don’t think we did… 😬_ it’s being instantiated with It gets it’s name from

__name__

Kevin Kho

08/04/2021, 3:56 PM

Yeah that would be the module name. Do you see the logs when you run on local with

flow.run()

Alexander van Eck

08/04/2021, 3:59 PM

The DockerAgent I’m also running is showing them yes

Alexander van Eck

08/04/2021, 4:01 PM

Copy code

Aug  4 16:00:50 workflow-01 prefect[10697]: [2021-08-04 16:00:41+0000] DEBUG - dask_jobqueue.core | Starting job: 54253

Alexander van Eck

08/04/2021, 4:04 PM

I’m not sure why --show-flow-logs in the agent and the Prefect UI wouldn’t show the same logs given that the level is debug in both.

Kevin Kho

08/04/2021, 4:09 PM

If you didn’t have the env variable, it wouldn’t even show up on the agent right?

Alexander van Eck

08/04/2021, 4:31 PM

Correct - I think it might be because our server is

0.14.21

and the agent is

0.15.3

Alexander van Eck

08/04/2021, 4:32 PM

do you know?

Kevin Kho

08/04/2021, 4:35 PM

I don’t have ideas yet, but I don’t think it’s that. What is your executor?

Alexander van Eck

08/04/2021, 4:51 PM

DaskExecutor

Kevin Kho

08/04/2021, 4:54 PM

Oh yeah of course 🤦‍♂️. Let me ask the team for any ideas and I’ll get back to you when there are.

Kevin Kho

08/04/2021, 4:57 PM

Wait sorry just wanna be sure I have all the details right. You see the debug level logs on the agent correctly, but not on the UI, but you see other debug level logs on the UI right?

Alexander van Eck

08/05/2021, 1:35 PM

@Kevin Kho Correct - I see the debug logs I would expect in the Agent logs but not in the UI. I see only prefect debug messages, specifically only

Checking flow run state...

Kevin Kho

08/05/2021, 2:44 PM

Unfortunately, I have no news for you yet. Will follow up with the team.

Alexander van Eck

08/17/2021, 3:23 PM

@Kevin Kho 👋 How’ve you been? 🙂 Any word on this issue?

Kevin Kho

08/17/2021, 3:26 PM

Hey, I have no ideas on this issue.

Kevin Kho

08/17/2021, 3:28 PM

The only thing I can possibly think of is that you might have the UI set to

INFO

logs? Do you see the filter in the UI? If it’s displaying on the agent with --show-flow-logs, it should be in the UI. Can you show me what the agent logs look like? I wanna see if it’s formatted like the Prefect logger.

Kevin Kho

08/17/2021, 3:34 PM

Went through this again. Looks like you have the UI not filtering logs because you saw the DEBUG logs. Could you try a LocalDaskExecutor perhaps and see if that makes it to the UI?

Kevin Kho

08/17/2021, 3:38 PM

Actually I think I have time today let me run a test.

Alexander van Eck

08/17/2021, 3:39 PM

Thank you 🙂 Unfortunately we are in different timezones and I’m just about to stop for the day. I’ll look into this tomorrow (Central European timezone)

Alexander van Eck

08/17/2021, 3:39 PM

I look forward to anything you find 🙌 and I’ll try to gather more info as per your comments

Kevin Kho

08/17/2021, 3:39 PM

Yeah I’ll give this a shot myself

Kevin Kho

08/18/2021, 1:19 PM

So I talked to the team, and the likely reason for this behavior is that the

dask-jobqueue

is running on a different process than the flow. because of that, the CloudHandler is not getting attached to the logger, so I’m not sure

dask-jobqueue

logs can easily be brought to the Prefect UI

2 Views

Open in Slack

Previous Next