https://prefect.io logo
Powered by
Title
b

Braun Reyes

08/26/2019, 5:42 AM
hello...newbie question...why doesn't flow failure return a non zero exit code. If one were to run this self contained inside of a docker container without using prefect cloud to start...how do you get the process to exit the docker run with error
c

Chris White

08/26/2019, 2:32 PM
Hey @Braun Reyes - one of Prefect’s features is error capturing and handling; 
flow.run()
will always only ever return a 
State
object corresponding to the flow’s final state. It sounds like you want non-zero exit codes for the sake of monitoring, which is a separate business - I’d recommend placing a terminal task on your Flow with an 
any_failed
trigger that calls 
sys.exit(1)
(or 
os._exit(1)
) to achieve your goal
j

Jeremiah

08/26/2019, 2:33 PM
It’s a good question — sometimes we call this “failing successfully.” What that means is Prefect tries not to apply business logic to your states (failures don’t take down the system). You’re more than welcome to add that logic yourself, however!
b

Braun Reyes

08/26/2019, 2:34 PM
is that how most companies accomplish the docker deployment use cases. In a world without prefect cloud, lets say you schedule your prefect flow as a simple docker task(ECS or Kube job), in my experience you want a non zero exit code on exception so that trickles down the typical monitoring and notification workflows
it sounds like I answered my own question...your explanations make sense.
I am pretty new to contributing to open source...I adapted the ShellTask to do exactly what the Airflow Bash Operator does...which streams the logs instead of outputting at the end. Would prefect be open to that kind of change? 
sub_process = Popen(
                [self.shell, tmp.name],
                stdout=PIPE,
                stderr=STDOUT,
                env=current_env
            )
            line = ''
            for raw_line in iter(sub_process.stdout.readline, b''):
                line = raw_line.decode('utf-8').rstrip()
                <http://self.logger.info|self.logger.info>(line)
            sub_process.wait()
            if sub_process.returncode:
                msg = "Command failed with exit code {0}: {1}".format(
                    sub_process.returncode, line
                )
                raise prefect.engine.signals.FAIL(msg) from None  # type: ignore
        return line
this was to support dbt commands in a way that feels seamless for how people currently use it
c

Chris White

08/26/2019, 2:42 PM
yea absolutely - we’d love a PR like this!
j

Jeremiah

08/26/2019, 2:49 PM
💯
#prefect-communityJoin Slack
Powered by