Hi I would like to use Prefect 2 0 to schedule run Java prog

Question

Hi I would like to use Prefect 2 0 to schedule run Java programs These programs as well as the shell scripts that invoke them produce stdout stderr messages What is the best practice for capturing stdout stderr from shell and non Python programs such that they appear in the Orion UI

kwmiebach · Answer

Hi < Araik Grigoryan> there may be several ways For example I am not sure if prefect offers something One thing that is quite popular in python is using the subprocess module and use pipes for communication I am assuming you are on linux or mac I am not sure how this would work on the other platforms

kwmiebach · Answer

Here is a simple example in this SO answer <https stackoverflow com a 72101287 > The question and the other answer might not really fit your use case but I believe this answer does In the example a git command is called but in your case the java application would have to be called The output of the called application is read in a loop In the example it is printed but you could also analyze and parse the output of the other process and do something more sophisticated with it And maybe you want to tell us your python version and also operating system where the python and java applications run

Araik Grigoryan · Answer

Hi < kwmiebach> I develop on the Mac but deploy to Linux I use Python 3 9 7 I am familiar with subprocess in Python so will give your suggestion a try Thank you

kwmiebach · Answer

There is some interesting code in this issue <https github com PrefectHQ prefect issues 2644> I don t know if it is prefect 1

Araik Grigoryan · Answer

< kwmiebach> Here s a solution adapted from your last suggested post that works in Prefect 2 It s not a complete solution because it logs all messages at `INFO` log level whereas a real external program will have messages with a variety of log levels So one has to do some extra work by parsing each of the `decoded line`s to match external program s log level with Prefect s log level This is the kind of thing that I would expect Prefect to do out of the box Thank you for the suggestion I hope this helps someone in the future

kwmiebach · Answer

< Araik Grigoryan> You could also check the return code and if it is an error code write that code to the prefect logger as `ERROR` If the java program has a list what the codes mean you could include the meaning in the log And maybe your java program does not only write to `stdout` but also to `stderr` After some modifcation in the python code you could also capture `stderr` in the python code and write the output if any to prefect as `ERROR `