When calling `future get state ` a 403 timeout response will

Question

When calling `future get state ` a 403 timeout response will cause flows to fail is there a way to see api limits for prefect orion cloud We get forbidden 403 occasionally it appears to be due to throttling

Malthe Karbo · Answer

we have implemented some internal throttling and exponential backoff it appears to solve it still would be interested in knowing the limits

Zanie · Answer

There should not be any throttling on this route we can investigate the 403 you are receiving

Malthe Karbo · Answer

Would love that please reach out if you need more information

Zanie · Answer

If you can reproduce it on a local server that d be helpful

Malthe Karbo · Answer

Cant We got it consistently on cloud though

Malthe Karbo · Answer

By that I mean we can t replicate it on local server

Malthe Karbo · Answer

< Zanie> did you figure out anything on your end

Zanie · Answer

I am working on other things right now I ve forwarded this to our team that s working on this but there s nothing to report yet

Zanie · Answer

We don t seem to have logs of the 403 response internally Can you share a couple timestamps A full traceback would be useful as well

Zach Angell · Answer

Hi < Malthe Karbo> thanks for reporting this We re looking into it on our end Do you have a sense for how often this happens when running against Cloud For example does it happen reliably each flow run Does the flow run have to make a large number of requests e g `future get state` calling `GET task runs id ` many times And yes as Michael mentioned traceback and timestamps would be helpful if you can share them

Malthe Karbo · Answer

It doesn t happen anymore after we implemented backoff in `future get state ` unfortunately we don t have the logs as we don t keep them in dev It was consistently failing almost every flow can you check traces logs on your end for our account

Malthe Karbo · Answer

We are now also receiving `ConnecTimeout` when running `future get result ` with exponential back off 5 retries ```Caught exception trying to get state from prefect api RetryError lt Future at 0x7fcfbe533b80 state=finished raised ConnectTimeout gt Traceback most recent call last File pipelines venv lib python3 9 site packages anyio core sockets py line 186 in connect tcp addr obj = ip address remote host File usr local lib python3 9 ipaddress py line 53 in ip address raise ValueError f address r does not appear to be an IPv4 or IPv6 address ValueError <http api beta prefect io|api beta prefect io> does not appear to be an IPv4 or IPv6 address During handling of the above exception another exception occurred Traceback most recent call last File pipelines venv lib python3 9 site packages httpcore backends asyncio py line 101 in connect tcp stream anyio abc ByteStream = await anyio connect tcp File pipelines venv lib python3 9 site packages anyio core sockets py line 189 in connect tcp gai res = await getaddrinfo asyncio exceptions CancelledError During handling of the above exception another exception occurred Traceback most recent call last File pipelines venv lib python3 9 site packages httpcore exceptions py line 8 in map exceptions yield File pipelines venv lib python3 9 site packages httpcore backends asyncio py line 101 in connect tcp stream anyio abc ByteStream = await anyio connect tcp File pipelines venv lib python3 9 site packages anyio core tasks py line 118 in exit raise TimeoutError TimeoutError During handling of the above exception another exception occurred Traceback most recent call last File pipelines venv lib python3 9 site packages httpx transports default py line 60 in map httpcore exceptions yield File pipelines venv lib python3 9 site packages httpx transports default py line 353 in handle async request resp = await self pool handle async request req File pipelines venv lib python3 9 site packages httpcore async connection pool py line 253 in handle async request raise exc File pipelines venv lib python3 9 site packages httpcore async connection pool py line 237 in handle async request response = await connection handle async request request File pipelines venv lib python3 9 site packages httpcore async connection py line 86 in handle async request raise exc File pipelines venv lib python3 9 site packages httpcore async connection py line 63 in handle async request stream = await self connect request File pipelines venv lib python3 9 site packages httpcore async connection py line 111 in connect stream = await self network backend connect tcp kwargs File pipelines venv lib python3 9 site packages httpcore backends auto py line 23 in connect tcp return await self backend connect tcp File pipelines venv lib python3 9 site packages httpcore backends asyncio py line 101 in connect tcp stream anyio abc ByteStream = await anyio connect tcp File usr local lib python3 9 contextlib py line 137 in exit self gen throw typ value traceback File pipelines venv lib python3 9 site packages httpcore exceptions py line 12 in map exceptions raise to exc exc httpcore ConnectTimeout The above exception was the direct cause of the following exception Traceback most recent call last File pipelines venv lib python3 9 site packages tenacity init py line 407 in call result = fn args kwargs File pipelines io utils py line 47 in check state state = future get state File pipelines venv lib python3 9 site packages prefect futures py line 243 in get state return cast State R sync self get state client=client File pipelines venv lib python3 9 site packages prefect utilities asyncio py line 150 in sync return run async from worker thread async fn args kwargs File pipelines venv lib python3 9 site packages prefect utilities asyncio py line 65 in run async from worker thread return anyio from thread run call File pipelines venv lib python3 9 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File pipelines venv lib python3 9 site packages anyio backends asyncio py line 970 in run async from thread return f result File usr local lib python3 9 concurrent futures base py line 446 in result return self get result File usr local lib python3 9 concurrent futures base py line 391 in get result raise self exception File pipelines venv lib python3 9 site packages prefect client py line 95 in with injected client return await fn args kwargs File pipelines venv lib python3 9 site packages prefect futures py line 249 in get state task run = await client read task run self run id File pipelines venv lib python3 9 site packages prefect client py line 1498 in read task run response = await self client get f task runs task run id File pipelines venv lib python3 9 site packages httpx client py line 1729 in get return await self request File pipelines venv lib python3 9 site packages httpx client py line 1506 in request return await self send request auth=auth follow redirects=follow redirects File pipelines venv lib python3 9 site packages prefect client py line 212 in send response = await super send args kwargs File pipelines venv lib python3 9 site packages httpx client py line 1593 in send response = await self send handling auth File pipelines venv lib python3 9 site packages httpx client py line 1621 in send handling auth response = await self send handling redirects File pipelines venv lib python3 9 site packages httpx client py line 1658 in send handling redirects response = await self send single request request File pipelines venv lib python3 9 site packages httpx client py line 1695 in send single request response = await transport handle async request request File pipelines venv lib python3 9 site packages httpx transports default py line 353 in handle async request resp = await self pool handle async request req File usr local lib python3 9 contextlib py line 137 in exit self gen throw typ value traceback File pipelines venv lib python3 9 site packages httpx transports default py line 77 in map httpcore exceptions raise mapped exc message from exc httpx ConnectTimeout The above exception was the direct cause of the following exception Traceback most recent call last File pipelines io utils py line 61 in as completed if check state future File pipelines venv lib python3 9 site packages tenacity init py line 324 in wrapped f return self f args kw File pipelines venv lib python3 9 site packages tenacity init py line 404 in call do = self iter retry state=retry state File pipelines venv lib python3 9 site packages tenacity init py line 361 in iter raise retry exc from fut exception tenacity RetryError RetryError lt Future at 0x7fcfbe533b80 state=finished raised ConnectTimeout gt ``` I can t share the flow code in more detail on here but we use the implementation from the as completed thread here on slack will paste below

Malthe Karbo · Answer

```from typing import Any TypeVar Iterable Generator from prefect futures import PrefectFuture from prefect utilities asyncio import Sync from tenacity import retry from tenacity stop import stop after attempt from tenacity wait import wait exponential T = TypeVar T retry stop=stop after attempt 5 wait=wait exponential multiplier=1 min=0 max=30 def check state future PrefectFuture Any Sync gt bool state = future get state return state is completed def as completed futures Iterable PrefectFuture T Sync gt Generator PrefectFuture T Sync None None logger = get run logger futures = set futures completed = set while len completed lt len futures for future in futures difference completed time sleep 0 1 try if check state future completed add future yield future except Exception as e logger exception f Caught exception trying to get state from prefect api e raise e``` This is the implementation used for as completed and ```from typing import TypeVar from tenacity import retry from tenacity wait import wait exponential from tenacity stop import stop after attempt from prefect futures import PrefectFuture from prefect utilities asyncio import Sync T = TypeVar T retry stop=stop after attempt 5 wait=wait exponential multiplier=1 min=0 max=30 def future result future PrefectFuture T Sync gt T return future result retry stop=stop after attempt 5 wait=wait exponential multiplier=1 min=0 max=30 def future wait future PrefectFuture T Sync future wait ``` These are the small utils for wait and direct result fetching used in flows We use these to wrap futures from tasks

Malthe Karbo · Answer

I understand that this issue is not the same as the 403 PermissionDenied but I figure they might be related

Malthe Karbo · Answer

We also receive runtime errors raised by httpx consistently like this one ```Encountered exception during execution Traceback most recent call last File pipelines venv lib python3 9 site packages tenacity init py line 407 in call result = fn args kwargs File tmp flow script lt gt flowg8kh63xy py line 48 in future result File pipelines venv lib python3 9 site packages prefect futures py line 208 in result return sync File pipelines venv lib python3 9 site packages prefect utilities asyncio py line 150 in sync return run async from worker thread async fn args kwargs File pipelines venv lib python3 9 site packages prefect utilities asyncio py line 65 in run async from worker thread return anyio from thread run call File pipelines venv lib python3 9 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File pipelines venv lib python3 9 site packages anyio backends asyncio py line 970 in run async from thread return f result File usr local lib python3 9 concurrent futures base py line 446 in result return self get result File usr local lib python3 9 concurrent futures base py line 391 in get result raise self exception File pipelines venv lib python3 9 site packages prefect futures py line 219 in result return final state result raise on failure=raise on failure File pipelines venv lib python3 9 site packages prefect orion schemas states py line 138 in result raise data File pipelines venv lib python3 9 site packages tenacity init py line 407 in call result = fn args kwargs File tmp flow script lt gt flowg8kh63xy py line 48 in future result File pipelines venv lib python3 9 site packages prefect futures py line 208 in result return sync File pipelines venv lib python3 9 site packages prefect utilities asyncio py line 150 in sync return run async from worker thread async fn args kwargs File pipelines venv lib python3 9 site packages prefect utilities asyncio py line 65 in run async from worker thread return anyio from thread run call File pipelines venv lib python3 9 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File pipelines venv lib python3 9 site packages anyio backends asyncio py line 970 in run async from thread return f result File usr local lib python3 9 concurrent futures base py line 446 in result return self get result File usr local lib python3 9 concurrent futures base py line 391 in get result raise self exception File pipelines venv lib python3 9 site packages prefect futures py line 219 in result return final state result raise on failure=raise on failure File pipelines venv lib python3 9 site packages prefect orion schemas states py line 138 in result raise data File pipelines venv lib python3 9 site packages tenacity init py line 407 in call result = fn args kwargs File tmp flow script lt gt flowg8kh63xy py line 48 in future result File pipelines venv lib python3 9 site packages prefect futures py line 208 in result return sync File pipelines venv lib python3 9 site packages prefect utilities asyncio py line 150 in sync return run async from worker thread async fn args kwargs File pipelines venv lib python3 9 site packages prefect utilities asyncio py line 65 in run async from worker thread return anyio from thread run call File pipelines venv lib python3 9 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File pipelines venv lib python3 9 site packages anyio backends asyncio py line 970 in run async from thread return f result File usr local lib python3 9 concurrent futures base py line 446 in result return self get result File usr local lib python3 9 concurrent futures base py line 391 in get result raise self exception File pipelines venv lib python3 9 site packages prefect futures py line 219 in result return final state result raise on failure=raise on failure File pipelines venv lib python3 9 site packages prefect orion schemas states py line 138 in result raise data File pipelines venv lib python3 9 site packages tenacity init py line 407 in call result = fn args kwargs File tmp flow script lt gt flowg8kh63xy py line 48 in future result File pipelines venv lib python3 9 site packages prefect futures py line 208 in result return sync File pipelines venv lib python3 9 site packages prefect utilities asyncio py line 150 in sync return run async from worker thread async fn args kwargs File pipelines venv lib python3 9 site packages prefect utilities asyncio py line 65 in run async from worker thread return anyio from thread run call File pipelines venv lib python3 9 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File pipelines venv lib python3 9 site packages anyio backends asyncio py line 970 in run async from thread return f result File usr local lib python3 9 concurrent futures base py line 446 in result return self get result File usr local lib python3 9 concurrent futures base py line 391 in get result raise self exception File pipelines venv lib python3 9 site packages prefect futures py line 219 in result return final state result raise on failure=raise on failure File pipelines venv lib python3 9 site packages prefect orion schemas states py line 138 in result raise data File pipelines venv lib python3 9 site packages tenacity init py line 407 in call result = fn args kwargs File tmp flow script lt gt flowg8kh63xy py line 48 in future result File pipelines venv lib python3 9 site packages prefect futures py line 208 in result return sync File pipelines venv lib python3 9 site packages prefect utilities asyncio py line 150 in sync return run async from worker thread async fn args kwargs File pipelines venv lib python3 9 site packages prefect utilities asyncio py line 65 in run async from worker thread return anyio from thread run call File pipelines venv lib python3 9 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File pipelines venv lib python3 9 site packages anyio backends asyncio py line 970 in run async from thread return f result File usr local lib python3 9 concurrent futures base py line 446 in result return self get result File usr local lib python3 9 concurrent futures base py line 391 in get result raise self exception File pipelines venv lib python3 9 site packages prefect futures py line 219 in result return final state result raise on failure=raise on failure File pipelines venv lib python3 9 site packages prefect orion schemas states py line 138 in result raise data File pipelines venv lib python3 9 site packages prefect task runners py line 412 in wait return await future result timeout=timeout File pipelines venv lib python3 9 site packages distributed client py line 294 in result raise exc with traceback tb File pipelines venv lib python3 9 site packages prefect engine py line 716 in begin task run return task run state File usr local lib python3 9 contextlib py line 670 in aexit raise exc details 1 File usr local lib python3 9 contextlib py line 653 in aexit cb suppress = await cb exc details File pipelines venv lib python3 9 site packages prefect client py line 1824 in aexit return await self exit stack aexit exc info File usr local lib python3 9 contextlib py line 670 in aexit raise exc details 1 File usr local lib python3 9 contextlib py line 653 in aexit cb suppress = await cb exc details File pipelines venv lib python3 9 site packages httpx client py line 1975 in aexit await self transport aexit exc type exc value traceback File pipelines venv lib python3 9 site packages httpx transports default py line 332 in aexit await self pool aexit exc type exc value traceback File pipelines venv lib python3 9 site packages httpcore async connection pool py line 326 in aexit await self aclose File pipelines venv lib python3 9 site packages httpcore async connection pool py line 312 in aclose raise RuntimeError RuntimeError The connection pool was closed while 1 HTTP requests responses were still in flight The above exception was the direct cause of the following exception Traceback most recent call last File pipelines venv lib python3 9 site packages prefect engine py line 468 in orchestrate flow run result = await run sync in worker thread flow call File pipelines venv lib python3 9 site packages prefect utilities asyncio py line 54 in run sync in worker thread return await anyio to thread run sync call cancellable=True File pipelines venv lib python3 9 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File pipelines venv lib python3 9 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File pipelines venv lib python3 9 site packages anyio backends asyncio py line 867 in run result = context run func args File tmp flow script lt gt flowg8kh63xy py line 222 in lt gt flow File pipelines venv lib python3 9 site packages tenacity init py line 324 in wrapped f return self f args kw File pipelines venv lib python3 9 site packages tenacity init py line 404 in call do = self iter retry state=retry state File pipelines venv lib python3 9 site packages tenacity init py line 361 in iter raise retry exc from fut exception tenacity RetryError RetryError lt Future at 0x7f22b779c430 state=finished raised RuntimeError gt ```

Malthe Karbo · Answer

Hey did you guys find out anything on your end

Zanie · Answer

Hi we re working a ton of things right now and we re a small team We ll report back when we find something I d recommend opening an issue on GitHub for long term tracking