hey prefect team, we had a flow crash overnight with a 404 error. using prefect cloud UI and prefect agent is v2.7.4. it has run hundreds of times before with no error like this, and I haven't changed any configuration settings. any ideas on what the cause could be for the bad api URL and how to prevent it in the future?

httpx.HTTPStatusError: Client error '404 Not Found' for url '<https://api.prefect.cloud/api/accounts/8ff05591-b922-43d6-96e2-0a956c1a55e7/workspaces/c573eb73-711f-42a5-a82d-f11e6defcd82/flow_runs/2c132385-8d43-471c-9dc5-dfad65eb5e41/set_state>'

Encountered exception during execution:
Traceback (most recent call last):
  File "/home/analytics/miniconda3/envs/prefect_env/lib/python3.10/site-packages/prefect/client/orion.py", line 1564, in set_flow_run_state
    response = await <http://self._client.post|self._client.post>(
  File "/home/analytics/miniconda3/envs/prefect_env/lib/python3.10/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
  File "/home/analytics/miniconda3/envs/prefect_env/lib/python3.10/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/analytics/miniconda3/envs/prefect_env/lib/python3.10/site-packages/prefect/client/base.py", line 251, in send
    response.raise_for_status()
  File "/home/analytics/miniconda3/envs/prefect_env/lib/python3.10/site-packages/httpx/_models.py", line 745, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url '<https://api.prefect.cloud/api/accounts/8ff05591-b922-43d6-96e2-0a956c1a55e7/workspaces/c573eb73-711f-42a5-a82d-f11e6defcd82/flow_runs/2c132385-8d43-471c-9dc5-dfad65eb5e41/set_state>'
For more information check: <https://httpstatuses.com/404>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/analytics/miniconda3/envs/prefect_env/lib/python3.10/site-packages/prefect/engine.py", line 1574, in report_flow_run_crashes
    yield
  File "/home/analytics/miniconda3/envs/prefect_env/lib/python3.10/site-packages/prefect/engine.py", line 514, in create_and_begin_subflow_run
    terminal_state = await orchestrate_flow_run(
  File "/home/analytics/miniconda3/envs/prefect_env/lib/python3.10/site-packages/prefect/engine.py", line 599, in orchestrate_flow_run
    state = await propose_state(client, Running(), flow_run_id=flow_run.id)
  File "/home/analytics/miniconda3/envs/prefect_env/lib/python3.10/site-packages/prefect/engine.py", line 1741, in propose_state
    response = await client.set_flow_run_state(
  File "/home/analytics/miniconda3/envs/prefect_env/lib/python3.10/site-packages/prefect/client/orion.py", line 1570, in set_flow_run_state
    raise prefect.exceptions.ObjectNotFound(http_exc=e) from e
prefect.exceptions.ObjectNotFound

Hi @David Cheedie have you had flows run successfully since this error? You may want to try updating your agent to v2.7.6 as well

Hi @Kalise Richmond, thanks for the response. The crash from last night was the only issue we had, there were no other flow run issues. And we've had consistently successful flow runs since then. Checked API key and it's not expired. I'm updating to prefect to v.2.7.6 now in case it was related to that

Let me know if after you update and you're still seeing the 404 happen. I don't see anything that on our status.prefect.io for last night either.