g

    Gustavo Silva

    1 year ago
    I would like to ask how do I get more information to find the issue with a failed flow. In my logs I juts got
    Process PID 266 returned non-zero exit code
    Kyle Moon-Wright

    Kyle Moon-Wright

    1 year ago
    Hey @Gustavo Silva, You can certainly get more information by configuring more verbose logging by doing either of the following: • start the agent with 
    show_flow_logs=True
     if using 
    flow.run_agent()
     or 
    --show-flow-logs
     if starting from the CLI •
    set PREFECT__LOGGING__LEVEL="DEBUG"
     in the Agent environment I feel like I typically see this error when there is a mismatch between Prefect versions - an Agent should have a greater or equal version of Prefect than the flow version. Not saying this is the issue here, but might be worth looking into.
    g

    Gustavo Silva

    1 year ago
    @Kyle Moon-Wright Awesome I will do this right away.
    m

    Miguel

    1 year ago
    Thanks @Kyle Moon-Wright The flow logs were just saying:
    Starting task run...
    Eventually, we increase the memory of the agent, restarted the flow and it was able to complete so we suspect the process was killed by the OOM killer Is there a way to log the exit code?
    Kyle Moon-Wright

    Kyle Moon-Wright

    1 year ago
    Hey @Miguel, You can certainly configure the logger by adjusting the settings found in the Prefect config or even add supplementary loggers as needed, but these details will be specific to your system so it’s hard to say which is best suited for your needs.
    m

    Miguel

    1 year ago
    This would have to be done at the level of the agent.https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/local/agent.py#L115 I guess we can override the heartbeat function and include that
    Kyle Moon-Wright

    Kyle Moon-Wright

    1 year ago
    Yeah this is a tough one, the Agent will never reach into your infrastructure due to the Hybrid Model even with a Server deployment - so OOM can be tough to discern amongst the noise, infra logs will probably be the most helpful in these cases.