Josh Lowe

    Josh Lowe

    1 year ago
    Hey everyone! This morning we've noticed that almost ALL of our flows have started failing with this error.
    Message: [{'path': ['set_flow_run_states'], 'message': '[{\'extensions\': {\'path\': \'$\', \'code\': \'data-exception\'}, \'message\': \'invalid input syntax for type uuid: "3fca7934-6af8-45f7-aae4-ea6fec0339c9:FargateAgent:agent:<agent_name>"\'}]', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
    No changes have been made to our platform, and it looks like some flows are able to be run, but then will fail again on subsequent runs. Anyone seen anything like this before? FargateAgent is deprecated, and we have plans to update and move to ECS agent - but I haven't seen any updates in a while that might be causing this
    m

    Mariia Kerimova

    1 year ago
    Hello Josh! What version of Prefect are you using?
    Josh Lowe

    Josh Lowe

    1 year ago
    Core version 0.12.0 currently! 😬 We’ve been due for an update for a while now
    m

    Mariia Kerimova

    1 year ago
    Okay, let me find out what changes were made on our side, I'll get back to you shortly.
    One more question to narrow the scope. Did you try to restart the agent and run a flow?
    Josh Lowe

    Josh Lowe

    1 year ago
    Yup, and it worked for a few runs then started throwing that error again
    m

    Mariia Kerimova

    1 year ago
    Can you please set debug level logs for this agent and confirm that you this log
    Agent ID: 7DEF1AEF-32B5-4D39-A2CA-D9D6D63B7471
    ?
    Josh Lowe

    Josh Lowe

    1 year ago
    Yup that's the agent id:
    7def1aef-32b5-4d39-a2ca-d9d6d63b7471
    Kevin Dyer

    Kevin Dyer

    1 year ago
    Did you see anything at your end @Mariia Kerimova or are you waiting for us to increase the debug level?
    Sorry for the delay. I bit of a change of guard at our end as Josh had to go offline
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Kevin Dyer, we’ll probably have to respond to you tom but yes increasing the debug level and showing logs. That might help us.
    Kevin Dyer

    Kevin Dyer

    1 year ago
    Hey Kevin, I jumped in and does the agent have debug logging? https://docs.prefect.io/orchestration/agents/fargate.html
    Kevin Kho

    Kevin Kho

    1 year ago
    On the agent, set the environment variable
    PREFECT__LOGGING__LEVEL=DEBUG
    . Doc
    You can do it upon agent start like this
    Actually wait, I think that CLI is for later versions.
    Looks it like works for version 0.12. Old docs
    Kevin Dyer

    Kevin Dyer

    1 year ago
    So
    export PREFECT__LOGGING__LEVEL=Debug; start_agent.py
    Where the script has
    agent = FargateAgent(...)
    agent.start()
    Should do it? Eventually it seems to rely on this line: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/agent.py#L145
    Kevin Kho

    Kevin Kho

    1 year ago
    I think you want
    PREFECT__LOGGING__LEVEL=DEBUG
    . Not sure if capitalization matters but just to be safe. That should do it. I wouldn’t really read code from master since we’re on version 0.14+ now and you were on 0.12 so I’m not sure it’s reliable. But yes in this case I think that’s all good.
    Kevin Dyer

    Kevin Dyer

    1 year ago
    Sorry for being slow to come back on this. We restarted everything to fix the issue. I'll try remember to check in regarding anything we have learnt, including if the log level change show more details on the agent
    Kevin Kho

    Kevin Kho

    1 year ago
    You can use the CLI when starting up an agent to control log levels btw.
    prefect agent local start -log-level DEBUG
    . Doc here . The environment variable is not the easiest way to do it
    Kevin Dyer

    Kevin Dyer

    1 year ago
    Sorry, I said agent but we are using a
    agent.start()
    script. I'd like to dig further (and upgrade) if I can make the time
    Kevin Kho

    Kevin Kho

    1 year ago
    Oh my bad. Yes I guess env variable is the way to go.