Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi. I would like to use Prefect 2.0 to schedule/run Java programs. These programs, as well as the shell scripts that invoke them, produce stdout/stderr messages. What is the best practice for capturing stdout/stderr from shell and non-Python programs such that they appear in the Orion UI?

Hi <@U041MGY7U00> there may be several ways. For example I am not sure if prefect offers something. One thing that is quite popular in python is using the subprocess module and use pipes for communication, I am assuming you are on linux or mac, I am not sure how this would work on the other platforms.

Here is a simple example in this SO answer: <https://stackoverflow.com/a/72101287/> The question and the other answer might not really fit your use case, but I believe this answer does. In the example a git command is called, but in your case the java application would have to be called. The output of the called application is read in a loop. In the example it is printed, but you could also analyze and parse the output of the other process and do something more sophisticated with it.

 And maybe you want to tell us your python version and also operating system where the python and java applications run.

Hi <@U02NBSBNPU4>. I develop on the Mac but deploy to Linux. I use Python 3.9.7. I am familiar with subprocess in Python, so will give your suggestion a try. Thank you!

There is some interesting code in this issue <https://github.com/PrefectHQ/prefect/issues/2644> - I don't know if it is prefect 1

Redirecting external log to Prefect logger.py

<@U02NBSBNPU4> Here’s a solution, adapted from your last suggested post, that works in Prefect 2. It’s not a complete solution because it logs all messages at `INFO` log level, whereas a real external program will have messages with a variety of log levels. So one has to do some extra work by parsing each of the `decoded_line`s to match external program’s log level with Prefect’s log level. This is the kind of thing that I would expect Prefect to do out of the box. Thank you for the suggestion. I hope this helps someone in the future.

<@U041MGY7U00> You could also check the return code and if it is an error code write that code to the prefect logger as `ERROR` . If the java program has a list what the codes mean you could include the meaning in the log.  And maybe your java program does not only write to `stdout` but also to `stderr`. After some modifcation in the python code you could also capture `stderr` in the python code and write the output, if any, to prefect as `ERROR.`