Have anyone been able to log pyspark stderr to prefect log? Trying to track down some issues and it turns out that pyspark write to stderr for Info. I am still on prefect v1.
z
Zanie
01/24/2023, 6:16 PM
You could set
sys.stderr = sys.stdout
at the start of your flow 😬
I’m not sure if that would work, but it’s a possibility.
p
Peter Peter
01/24/2023, 6:17 PM
I tried that and it did not seem to work.
going to try again
z
Zanie
01/24/2023, 6:19 PM
It’s possible pyspark sets up its logging before you mutate it in which case there’s not much you can do. Is it in a subprocess?