Hi Team!
Is there a way we can get prefect logs to also pick up logs from a .txt file? What's essentially happening is that we are using Prefect to kick off another job, which logs to a .txt file, but we want prefect to pick up those logs in real time as the process executes. I understand there's a logging.yml, is there a parameter there that could help us?
c
Christopher Boyd
05/26/2023, 10:05 PM
You’d need to redirect the logs from the file to stdout / stderr but yes - this is basically all side car logging in kubernetes is
c
Charles Leung
05/26/2023, 11:35 PM
Thanks @Christopher Boyd! So basically, Prefect kicks off a flow which opens a port via IPC to kick off a process on that new port, and run a function that contains logging via stdout. However, the logging is only showing on just that other port, not on the prefect console. We want to figure out a way to relay those logs back to the original prefect process that kicked it off. Hopefully that makes sense!
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.