Hi. I would like to use Prefect 2.0 to schedule/run Java programs. These programs, as well as the sh...

Araik Grigoryan

09/03/2022, 1:15 AM

Hi. I would like to use Prefect 2.0 to schedule/run Java programs. These programs, as well as the shell scripts that invoke them, produce stdout/stderr messages. What is the best practice for capturing stdout/stderr from shell and non-Python programs such that they appear in the Orion UI?

✅ 1

kwmiebach

09/03/2022, 2:26 PM

Hi @Araik Grigoryan there may be several ways. For example I am not sure if prefect offers something. One thing that is quite popular in python is using the subprocess module and use pipes for communication, I am assuming you are on linux or mac, I am not sure how this would work on the other platforms.

👍 1

kwmiebach

09/03/2022, 2:34 PM

Here is a simple example in this SO answer: https://stackoverflow.com/a/72101287/ The question and the other answer might not really fit your use case, but I believe this answer does. In the example a git command is called, but in your case the java application would have to be called. The output of the called application is read in a loop. In the example it is printed, but you could also analyze and parse the output of the other process and do something more sophisticated with it. And maybe you want to tell us your python version and also operating system where the python and java applications run.

Araik Grigoryan

09/03/2022, 2:46 PM

Hi @kwmiebach. I develop on the Mac but deploy to Linux. I use Python 3.9.7. I am familiar with subprocess in Python, so will give your suggestion a try. Thank you!

kwmiebach

09/03/2022, 3:15 PM

There is some interesting code in this issue https://github.com/PrefectHQ/prefect/issues/2644 - I don't know if it is prefect 1

Araik Grigoryan

09/04/2022, 2:29 PM

@kwmiebach Here’s a solution, adapted from your last suggested post, that works in Prefect 2. It’s not a complete solution because it logs all messages at

INFO

log level, whereas a real external program will have messages with a variety of log levels. So one has to do some extra work by parsing each of the `decoded_line`s to match external program’s log level with Prefect’s log level. This is the kind of thing that I would expect Prefect to do out of the box. Thank you for the suggestion. I hope this helps someone in the future.

Redirecting external log to Prefect logger.py

kwmiebach

09/06/2022, 10:27 AM

@Araik Grigoryan You could also check the return code and if it is an error code write that code to the prefect logger as

ERROR

. If the java program has a list what the codes mean you could include the meaning in the log. And maybe your java program does not only write to

stdout

but also to

stderr

. After some modifcation in the python code you could also capture

stderr

in the python code and write the output, if any, to prefect as

ERROR.

21 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.