Hey everyone! This morning we've noticed that almost ALL of our flows have started failing with this error.

Message: [{'path': ['set_flow_run_states'], 'message': '[{\'extensions\': {\'path\': \'$\', \'code\': \'data-exception\'}, \'message\': \'invalid input syntax for type uuid: "3fca7934-6af8-45f7-aae4-ea6fec0339c9:FargateAgent:agent:<agent_name>"\'}]', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

No changes have been made to our platform, and it looks like some flows are able to be run, but then will fail again on subsequent runs. Anyone seen anything like this before? FargateAgent is deprecated, and we have plans to update and move to ECS agent - but I haven't seen any updates in a while that might be causing this

Hello Josh! What version of Prefect are you using?

Core version 0.12.0 currently! 😬 We’ve been due for an update for a while now

Okay, let me find out what changes were made on our side, I'll get back to you shortly.

Yup, and it worked for a few runs then started throwing that error again

Can you please set debug level logs for this agent and confirm that you this log

Agent ID: 7DEF1AEF-32B5-4D39-A2CA-D9D6D63B7471

?

Yup that's the agent id:

7def1aef-32b5-4d39-a2ca-d9d6d63b7471

Did you see anything at your end @Mariia Kerimova or are you waiting for us to increase the debug level?

Hey @Kevin Dyer, we’ll probably have to respond to you tom but yes increasing the debug level and showing logs. That might help us.

Hey Kevin, I jumped in and does the agent have debug logging? https://docs.prefect.io/orchestration/agents/fargate.html

On the agent, set the environment variable

PREFECT__LOGGING__LEVEL=DEBUG

. Doc

So

export PREFECT__LOGGING__LEVEL=Debug; start_agent.py

Where the script has

agent = FargateAgent(...)
agent.start()

Should do it? Eventually it seems to rely on this line: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/agent.py#L145

I think you want

PREFECT__LOGGING__LEVEL=DEBUG

. Not sure if capitalization matters but just to be safe. That should do it. I wouldn’t really read code from master since we’re on version 0.14+ now and you were on 0.12 so I’m not sure it’s reliable. But yes in this case I think that’s all good.

Sorry for being slow to come back on this. We restarted everything to fix the issue. I'll try remember to check in regarding anything we have learnt, including if the log level change show more details on the agent

You can use the CLI when starting up an agent to control log levels btw.

prefect agent local start -log-level DEBUG

. Doc here . The environment variable is not the easiest way to do it

Sorry, I said agent but we are using a

agent.start()

script. I'd like to dig further (and upgrade) if I can make the time

Oh my bad. Yes I guess env variable is the way to go.