Am seeing this exception while trying to run a simple flow:

File "/opt/conda/envs/coiled/lib/python3.9/site-packages/prefect/client/base.py", line 125, in raise_for_status
Exception: PrefectHTTPStatusError("Client error '404 Not Found' for url '<http://ephemeral-orion/api/task_runs/add94276-cb3e-417d-a3f9-033f935ae5ec/set_state>'\nResponse: {'exception_message': 'Task run with id add94276-cb3e-417d-a3f9-033f935ae5ec not found'}\nFor more information check: <https://httpstatuses.com/404>")

Any ideas as to what might cause this?

Glad you arrived at a solution!

are the API keys one-time use only?

Nope you can use them in whatever capacity you deem necessary as frequently as you wish, there is no way to retrieve it if you lose track of the key but they aren't limited to 1 time use.

i have attempted to reuse them but they never seem to work the 2nd time.

Are you seeing these same or similar errors in those cases, I can definitively say there's no limit on the number of time you can use an API Key, you can set an expiration for them when they're created, they can also be deleted but assuming the API key is still active through cloud and you still have access to it there's no limit on the number of times it's used.

pretty consistent behavior but not in the mood to test it now that my flows are working 😉

Fair Enough 😄 Some preliminary thoughts if/when you do run into these issues, is this part of a CI/CD process, are you setting the API_URL and API_KEY during that process, seeing this URL

<http://ephemeral-orion/api/task_runs/add94276-cb3e-417d-a3f9-033f935ae5ec/set_state>

indicates that the API is not pointing to a Cloud or Local Orion instance so it's generating an Ephemeral api instead so there may be something misconfigured during that setup.

mostly i have just logged in using prefect cloud login

For sure, we're always happy to help 😄

really appreciate the support of the prefect folks. you folks are awesome!