Hello :wave: I have gotten a few flows today that have failed when the state of the flow is trying t...

John Mizerany

07/11/2023, 9:18 PM

Hello 👋 I have gotten a few flows today that have failed when the state of the flow is trying to update. I’ll post the error in the thread

John Mizerany

07/11/2023, 9:18 PM

Copy code

State message: Flow run encountered an exception. PrefectHTTPStatusError: Server error '503 Service Unavailable' for url '<https://api.prefect.cloud/api/accounts/><account_id>/workspaces/<workspace>/task_runs/<task_run>/set_state'
Response: {'exception_message': 'Service Unavailable'}

John Mizerany

07/11/2023, 9:19 PM

I left out my account_id, workspace and task run id. It looks like issues with the Prefect api, but I was not sure if this was happening across other accounts

John Mizerany

07/12/2023, 3:06 PM

I’ve had this pop up a few more times today again, wanted to see if anyone else was experiencing this

Christopher Boyd

07/12/2023, 3:30 PM

Can you share a few flow run id’s and message me your account/workspace - I’m curious that no one else ahs reported this, but you’re still getting 503's from set_state

John Mizerany

07/12/2023, 4:17 PM

Hey! Yes I can dm you the info

✅ 1

Christopher Boyd

07/12/2023, 5:37 PM

Hi John, it looks like there are some invalid characters happening in your code, and it’s failing to write to teh cloud database

Christopher Boyd

07/12/2023, 5:37 PM

Copy code

(sqlalchemy.dialects.postgresql.asyncpg.Error) <class 'asyncpg.exceptions.CharacterNotInRepertoireError'>: invalid byte sequence for encoding "UTF8": 0x00

Christopher Boyd

07/12/2023, 5:37 PM

John Mizerany

07/12/2023, 5:38 PM

Ohhhh that looks like a duplicate DML from snowflake

John Mizerany

07/12/2023, 5:38 PM

Okay that makes more sense. Thank you

John Mizerany

07/12/2023, 5:38 PM

The logging in the prefect UI did not show me that so that helps me out a lot

Christopher Boyd

07/12/2023, 5:39 PM

Yea, I think this was a good call out all the same, we are debating if a 503 is the right return here

Christopher Boyd

07/12/2023, 5:39 PM

503 implies a server side error (which it is) but as a result of a bad input which was client input, so it threw me off here

John Mizerany

07/12/2023, 5:40 PM

Yep that’s what I thought it was an issue on prefects side, but now that I know it’s on our end I can fix the issue

✅ 1

John Mizerany

07/12/2023, 5:40 PM

Thanks!

🙌 1

2 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.