https://prefect.io logo
Title
g

gertjan

09/28/2022, 1:57 PM
Hi all. I’m able to stream my program’s logs to the UI setting this ENV variable:
export PREFECT_LOGGING_EXTRA_LOGGERS='my-loggers-name'
Now, when I run my codebase in multiprocessing these logs are not shown in the UI. Anyone has some tips how I still stream the logs of
my-loggers-name
to the UI while using multiprocessing? EDIT: the
root
Logger is also ignored..
r

Rob Freedy

09/29/2022, 1:40 PM
Are you able to set the logger as an environment variable in the run config or in the agent start command you are using like in this example? https://discourse.prefect.io/t/vertex-agent-extra-loggers/821/3
g

gertjan

09/29/2022, 1:50 PM
I’m on prefect 2.0, don’t think the agent has a
--env
var? Also, I can see already sone differences using
spawn
vs
fork
in multiprocessing, but still no luck to see these in the UI. (single process still works)
r

Rob Freedy

09/29/2022, 2:04 PM
ah apologies that was for 1.0. Are you able to set the extra loggers variable within the code like in this example: https://discourse.prefect.io/t/how-to-use-logging-in-prefect-a-tutorial-by-andrew-brookins/197/4#ray-3? Where are you setting that variable using the export statement?
g

gertjan

09/29/2022, 2:06 PM
Yes I am. In my prefect code I’m just using the default taks_runner. (No Dask or Ray). But in a task, I call a function that uses multiprocessing. I can see those logs in my console, but not the UI. When I run the same code, without multiprocessing, the logs are visible in the UI. Which means this
PREFECT_LOGGING_EXTRA_LOGGERS
is actually being detected, and the UI also shows the logs from my applications’s logger (or even the root logger). Riddle to solve, why are these logs outputted in my application’s multiprocessing code not visible? UPDATE: When I run my application multiprocessing on
spawn
I don’t see my applications logger in the console. When running on
fork
I see it in the console, but not in the UI
r

Rob Freedy

09/30/2022, 1:29 PM
Appreciate the information! I don't believe that the flow/task run context is shared between processes automatically, so you would have to pass a copy context through to each process. We will have our team take a look.