https://prefect.io logo
Title
p

Peter Peter

01/24/2023, 6:02 PM
Have anyone been able to log pyspark stderr to prefect log? Trying to track down some issues and it turns out that pyspark write to stderr for Info. I am still on prefect v1.
z

Zanie

01/24/2023, 6:16 PM
You could set
sys.stderr = sys.stdout
at the start of your flow 😬
I’m not sure if that would work, but it’s a possibility.
p

Peter Peter

01/24/2023, 6:17 PM
I tried that and it did not seem to work.
going to try again
z

Zanie

01/24/2023, 6:19 PM
It’s possible pyspark sets up its logging before you mutate it in which case there’s not much you can do. Is it in a subprocess?
p

Peter Peter

01/24/2023, 6:21 PM
not running via a subprocess directly