Hello! I am dealing with a logger problem, while I am using pyspark-kafka module with custom Streami...

Kacper K

11/25/2024, 12:14 PM

Hello! I am dealing with a logger problem, while I am using pyspark-kafka module with custom StreamingQueryListener I can't see any outputs/logs in FlowRun area for my run. Do you have any ideas how to add it? Thanks!

Nate

11/25/2024, 2:43 PM

hi @Kacper K - have you tried setting

PREFECT_LOGGING_EXTRA_LOGGERS=pyspark-kafka

? that will work if

pyspark-kafka

defines a custom python logger and thats what you want to capture

Kacper K

11/27/2024, 2:12 PM

Can I set PREFECT_LOGGING_EXTRA_LOGGERS to 2 values?

Kacper K

11/27/2024, 2:12 PM

Then I would add both custom logger and pyspark-kafka

Nate

11/27/2024, 2:28 PM

yes! it can be a comma delimited string

Copy code

» PREFECT_LOGGING_EXTRA_LOGGERS=foo,bar ipython

In [1]: from prefect.settings import PREFECT_LOGGING_EXTRA_LOGGERS

In [2]: PREFECT_LOGGING_EXTRA_LOGGERS.value()
Out[2]: ['bar', 'foo']

Kacper K

11/27/2024, 2:32 PM

Should I restart worker to see results?

Nate

11/27/2024, 2:34 PM

if its a process worker then yes, otherwise it shouldnt matter

Nate

11/27/2024, 2:35 PM

you'll want to set this where your flow code is running, which is not the actual worker process (unless its a process worker)

Kacper K

11/27/2024, 2:38 PM

it is quite werid I can see logs in worker screen, but not i UI, should I enable some function?

Kacper K

11/27/2024, 2:39 PM

This is a warning:

logger.py:53: UserWarning: Logger 'ds-data-logger' attempted to send logs to the API without a flow run id. The API log handler can only send logs within flow run contexts unless the flow run id is manually provided.

7 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.