Darren Fleetwood
06/24/2022, 6:07 AMPREFECT__LOGGING__EXTRA_LOGGERS= "['ray']"
As well as this within the flow:
prefect.config.logging.extra_loggers = ['ray']
Plus a few other things, none of which have worked. This is the case for all logging levels (error, info, etc.)
What’s the proper way of doing this?
Thanks!Kevin Kho
06/24/2022, 1:37 PMBy default, all of the tasks/actors stdout and stderr are redirected to the worker log files. Check out Logging directory structure to learn how Ray's logging directory is structured.
By default, all of the tasks/actors stdout and stderr that is redirected to worker log files are published to the driver. Drivers display logs generated from its tasks/actors to its stdout and stderr.
So I think the best attempt is to add
@task(log_stdout=True)
to the task along with the extra loggers, and hopefully the driver stdout can capture.
Also, doing
prefect.config.logging.extra_loggers = ['ray']
inside the flow will not work because the context has been instantiated already.Darren Fleetwood
06/24/2022, 1:54 PM@task(log_stdout=True)
but it's not coming through... though I can see them in the logs of the kubernetes pods they're running in.
As a work-around, I've tried explicitly using the prefect logger in the ray actors in the tasks with
logger = prefect.context.get("logger")
but that also doesn't seem to want to work (presumably this requires the prefect logger to be serialised somehow??). I'll give the ray guys a shout and see if they have any thoughtsKevin Kho
06/24/2022, 1:57 PM