A project Orion question, I seem to recall there was going to be support for wrangling long running tasks? I have changed company and we have applications that are started and run for a day, sometimes 3, but have a range of dependancies, at the moment in some crazy complicated shell scripts that serially start thing that can take a long time. I already wrote an app that starts many of them in parallel and monitors the output using filesystem watches but there is quite a complicated DAG of things to run. Many of them fork and exit and I am considering them done when I see a message in a file or journald _comm watch message. (They are not actually done, but other processes can be run then). Ideally I'd like to express the existing 5000 line shell script of a prefect flow.

Hey @Rob Fowler, congrats on the job change! On the first post, I feel like it can be done in current Prefect. We can have long running tasks. You can construct this as a Prefect flow using the ShellTask and reading the outputs of the ShellTask, and then doing those forks and exits. With regards to Orion, there is nothing there is supporting the long running task execution. But, it may still be a better fit because of the better handling of subflows, allowing you to have more modular code. That 5000 line shells script can likely be broken up much better. On the second comment, the Flow being a short running process, that can also work. On the Prefect side, we don’t care about execution time anyway. Just an idea, but you can maybe also use manual triggers to check on the status of Flows and trigger the continuation from day to day.

Thanks for the advice. I was very used to the flows in 'current' prefect but I would add, I often had people stumble on the way it worked in the context manager. The new system is more pythonic and it seems too easy now 🙂 Ye olde 'ShellTask' never survived more than the prototype and MCS but it was a good basis for lots of things. You are right about native subflows being a great improvement for this shell script replacement. I had many subflows before but they mandated having a full prefect server up and were never as simple as running things in the local command line for tests. The more I think about these long running processes the more I think they can be represented, at the very least, by two graphs, one for starting and one for stopping.