Have anyone been able to log pyspark stderr to prefect log? Trying to track down some issues and it turns out that pyspark write to stderr for Info. I am still on prefect v1.

You could set

sys.stderr = sys.stdout

at the start of your flow 😬

I tried that and it did not seem to work.

It’s possible pyspark sets up its logging before you mutate it in which case there’s not much you can do. Is it in a subprocess?

not running via a subprocess directly