Channels
#prefect-server
Title
a
Alexander van Eck
08/04/2021, 1:41 PMMorning 👋
I’m using
DaskExecutorcluster_class Copy code
15:36:12 INFO DaskExecutor Creating a new Dask cluster with `dask_jobqueue.remote_slurm.RemoteSLURMCluster.with_http_api_client`...
15:36:13 INFO DaskExecutor The Dask dashboard is available at <http://10.240.3.45:8787/status>k
Kevin Kho
08/04/2021, 1:47 PM
Hey @Alexander van Eck , morning!
I think you’re trying to add extra loggers like this?
a
Alexander van Eck
08/04/2021, 1:49 PMInteresting!
I suppose
Copy code
export PREFECT__LOGGING__EXTRA_LOGGERS="['']"k
Kevin Kho
08/04/2021, 1:55 PM
Yeah you can add it to the
RunConfigdask-jobquuePREFECT___LOGGING___LEVEL="DEBUG"a
Alexander van Eck
08/04/2021, 1:55 PMThanks 🙂 I’ll test this out
hm 🤔 @Kevin Kho
I did
Copy code
'PREFECT__LOGGING__EXTRA_LOGGERS': "['dask_jobqueue']",
'PREFECT__LOGGING__LEVEL': 'DEBUG',k
Kevin Kho
08/04/2021, 3:10 PM
This the debug level logs at least work?
a
Alexander van Eck
08/04/2021, 3:25 PMI can see prefect debug logs yes (
Checking flow run state...dask_jobqueuek
Kevin Kho
08/04/2021, 3:33 PM
Maybe we got the name of the logger wrong? Would you know the logger name?
a
Alexander van Eck
08/04/2021, 3:55 PMI don’t think we did… 😬_ it’s being instantiated with It gets it’s name from
__name__k
Kevin Kho
08/04/2021, 3:56 PM
Yeah that would be the module name. Do you see the logs when you run on local with
flow.run()a
Alexander van Eck
08/04/2021, 3:59 PMThe DockerAgent I’m also running is showing them yes
Copy code
Aug 4 16:00:50 workflow-01 prefect[10697]: [2021-08-04 16:00:41+0000] DEBUG - dask_jobqueue.core | Starting job: 54253I’m not sure why --show-flow-logs in the agent and the Prefect UI wouldn’t show the same logs given that the level is debug in both.
k
Kevin Kho
08/04/2021, 4:09 PM
If you didn’t have the env variable, it wouldn’t even show up on the agent right?
a
Alexander van Eck
08/04/2021, 4:31 PMCorrect - I think it might be because our server is
0.14.210.15.3do you know?
k
Kevin Kho
08/04/2021, 4:35 PM
I don’t have ideas yet, but I don’t think it’s that. What is your executor?
a
Alexander van Eck
08/04/2021, 4:51 PMDaskExecutor
k
Kevin Kho
08/04/2021, 4:54 PM
Oh yeah of course 🤦♂️. Let me ask the team for any ideas and I’ll get back to you when there are.
Wait sorry just wanna be sure I have all the details right. You see the debug level logs on the agent correctly, but not on the UI, but you see other debug level logs on the UI right?
a
Alexander van Eck
08/05/2021, 1:35 PM@Kevin Kho Correct - I see the debug logs I would expect in the Agent logs but not in the UI.
I see only prefect debug messages, specifically only
Checking flow run state...k
Kevin Kho
08/05/2021, 2:44 PM
Unfortunately, I have no news for you yet. Will follow up with the team.
a
Alexander van Eck
08/17/2021, 3:23 PM@Kevin Kho 👋 How’ve you been? 🙂 Any word on this issue?
k
Kevin Kho
08/17/2021, 3:26 PM
Hey, I have no ideas on this issue.
The only thing I can possibly think of is that you might have the UI set to
INFOWent through this again. Looks like you have the UI not filtering logs because you saw the DEBUG logs. Could you try a LocalDaskExecutor perhaps and see if that makes it to the UI?
Actually I think I have time today let me run a test.
a
Alexander van Eck
08/17/2021, 3:39 PMThank you 🙂 Unfortunately we are in different timezones and I’m just about to stop for the day. I’ll look into this tomorrow (Central European timezone)
I look forward to anything you find 🙌 and I’ll try to gather more info as per your comments
k
Kevin Kho
08/17/2021, 3:39 PM
Yeah I’ll give this a shot myself
So I talked to the team, and the likely reason for this behavior is that the
dask-jobqueuedask-jobqueue2 Views