Hi all. I have a bunch of Ray Actors and functions that are created within a Prefect flow. I don't seem to be able to access the Ray logs when the flow is run in Prefect Cloud (1.x). This works fine when I'm testing locally. I’ve tried setting this env var:

PREFECT__LOGGING__EXTRA_LOGGERS= "['ray']"

As well as this within the flow:

prefect.config.logging.extra_loggers = ['ray']

Plus a few other things, none of which have worked. This is the case for all logging levels (error, info, etc.) What’s the proper way of doing this? Thanks!

Hey Darren, this depends on ray logs. Driver logs are normally just the same as Python logs, and worker logs will take some work. From the Ray docs here:

By default, all of the tasks/actors stdout and stderr are redirected to the worker log files. Check out Logging directory structure to learn how Ray's logging directory is structured.

By default, all of the tasks/actors stdout and stderr that is redirected to worker log files are published to the driver. Drivers display logs generated from its tasks/actors to its stdout and stderr.

So I think the best attempt is to add

@task(log_stdout=True)

to the task along with the extra loggers, and hopefully the driver stdout can capture. Also, doing

prefect.config.logging.extra_loggers = ['ray']

inside the flow will not work because the context has been instantiated already.

Ok thanks Kevin. It's the worker logs I'm interested in, but as you linked to above they should be routed through to the driver anyway. I've tried

@task(log_stdout=True)

but it's not coming through... though I can see them in the logs of the kubernetes pods they're running in. As a work-around, I've tried explicitly using the prefect logger in the ray actors in the tasks with

logger = prefect.context.get("logger")

but that also doesn't seem to want to work (presumably this requires the prefect logger to be serialised somehow??). I'll give the ray guys a shout and see if they have any thoughts

I believe that doesn’t look because the logger is pickled to be sent to a worker, and then it loses configuration when unpickled. At least for Dask, that is the case, I expect it to be the same for ray