Hey everyone This morning we ve noticed that almost ALL of o

Question

Hey everyone This morning we ve noticed that almost ALL of our flows have started failing with this error ```Message path set flow run states message extensions path $ code data exception message invalid input syntax for type uuid 3fca7934 6af8 45f7 aae4 ea6fec0339c9 FargateAgent agent lt agent name gt extensions code INTERNAL SERVER ERROR ``` No changes have been made to our platform and it looks like some flows are able to be run but then will fail again on subsequent runs Anyone seen anything like this before FargateAgent is deprecated and we have plans to update and move to ECS agent but I haven t seen any updates in a while that might be causing this

Mariia Kerimova · Answer

Hello Josh What version of Prefect are you using

Josh Lowe · Answer

Core version 0 12 0 currently grimacing We ve been due for an update for a while now

Mariia Kerimova · Answer

Okay let me find out what changes were made on our side I ll get back to you shortly

Mariia Kerimova · Answer

One more question to narrow the scope Did you try to restart the agent and run a flow

Josh Lowe · Answer

Yup and it worked for a few runs then started throwing that error again

Mariia Kerimova · Answer

Can you please set debug level logs for this agent and confirm that you this log `Agent ID 7DEF1AEF 32B5 4D39 A2CA D9D6D63B7471`

Josh Lowe · Answer

Yup that s the agent id `7def1aef 32b5 4d39 a2ca d9d6d63b7471`

Kevin Dyer · Answer

Did you see anything at your end < Mariia Kerimova> or are you waiting for us to increase the debug level

Kevin Dyer · Answer

Sorry for the delay I bit of a change of guard at our end as Josh had to go offline

Kevin Kho · Answer

Hey < Kevin Dyer> we ll probably have to respond to you tom but yes increasing the debug level and showing logs That might help us

Kevin Dyer · Answer

Hey Kevin I jumped in and does the agent have debug logging <https docs prefect io orchestration agents fargate html>

Kevin Kho · Answer

On the agent set the environment variable `PREFECT LOGGING LEVEL=DEBUG` <https docs prefect io core concepts logging html logging configuration|Doc>

Kevin Kho · Answer

You can do it upon agent start like <https docs prefect io orchestration agents overview html environment variables|this>

Kevin Kho · Answer

Actually wait I think that CLI is for later versions

Kevin Kho · Answer

Looks it like works for version 0 12 <https docs prefect io api 0 12 6 agent fargate html fargateagent|Old docs>

Kevin Dyer · Answer

So `export PREFECT LOGGING LEVEL=Debug start agent py` Where the script has ```agent = FargateAgent agent start ``` Should do it Eventually it seems to rely on this line <https github com PrefectHQ prefect blob master src prefect agent agent py L145>

Kevin Kho · Answer

I think you want `PREFECT LOGGING LEVEL=DEBUG` Not sure if capitalization matters but just to be safe That should do it I wouldn t really read code from master since we re on version 0 14+ now and you were on 0 12 so I m not sure it s reliable But yes in this case I think that s all good

Kevin Dyer · Answer

Sorry for being slow to come back on this We restarted everything to fix the issue I ll try remember to check in regarding anything we have learnt including if the log level change show more details on the agent

Kevin Kho · Answer

You can use the CLI when starting up an agent to control log levels btw `prefect agent local start log level DEBUG` <https docs prefect io api latest cli agent html agent|Doc here> The environment variable is not the easiest way to do it

Kevin Dyer · Answer

Sorry I said agent but we are using a `agent start ` script I d like to dig further and upgrade if I can make the time

Kevin Kho · Answer

Oh my bad Yes I guess env variable is the way to go