Hey all When using `task map` on a large sequence of inputs

Question

Hey all When using `task map` on a large sequence of inputs while defining a tag concurrency limit of 20 prefect seems to crash a lot while trying `set state` of tasks with a status code 500 traceback in the thread arrow down Do you have any recommandations on how to work around this can define retries for prefect internal API is it something I m doing wrong

Ouail Bendidi · Answer

``` File home obendidi pyenv versions 3 11 2 lib python3 11 asyncio runners py line 190 in run return runner run main ^^^^^^^^^^^^^^^^ File home obendidi pyenv versions 3 11 2 lib python3 11 asyncio runners py line 118 in run return self loop run until complete task ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File home obendidi pyenv versions 3 11 2 lib python3 11 asyncio base events py line 653 in run until complete return future result ^^^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect internal concurrency api py line 109 in wait for call in loop thread return call result ^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect internal concurrency calls py line 173 in result return self future result timeout=timeout ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File home obendidi pyenv versions 3 11 2 lib python3 11 concurrent futures base py line 449 in result return self get result ^^^^^^^^^^^^^^^^^^^ File home obendidi pyenv versions 3 11 2 lib python3 11 concurrent futures base py line 401 in get result raise self exception File home user myproject venv lib python3 11 site packages prefect internal concurrency calls py line 218 in run async result = await coro ^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect client utilities py line 40 in with injected client return await fn args kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect engine py line 259 in create then begin flow run return await state result fetch=True ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect states py line 91 in get state result raise await get state exception state File home user myproject venv lib python3 11 site packages prefect engine py line 674 in orchestrate flow run result = await flow call aresult ^^^^^^^^^^^^^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect internal concurrency calls py line 181 in aresult return await asyncio wrap future self future ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect internal concurrency calls py line 218 in run async result = await coro ^^^^^^^^^^ File home user myproject flows sync kili labels flow py line 70 in sync kili labels flow async with anyio create task group as tg File home user myproject venv lib python3 11 site packages anyio backends asyncio py line 662 in aexit raise exceptions 0 File home user myproject venv lib python3 11 site packages prefect internal concurrency api py line 109 in wait for call in loop thread return call result ^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect internal concurrency calls py line 173 in result return self future result timeout=timeout ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File home obendidi pyenv versions 3 11 2 lib python3 11 concurrent futures base py line 449 in result return self get result ^^^^^^^^^^^^^^^^^^^ File home obendidi pyenv versions 3 11 2 lib python3 11 concurrent futures base py line 401 in get result raise self exception File home user myproject venv lib python3 11 site packages prefect internal concurrency calls py line 218 in run async result = await coro ^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect client utilities py line 40 in with injected client return await fn args kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect engine py line 571 in create and begin subflow run return await terminal state result fetch=True ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect states py line 91 in get state result raise await get state exception state File home user myproject venv lib python3 11 site packages prefect engine py line 674 in orchestrate flow run result = await flow call aresult ^^^^^^^^^^^^^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect internal concurrency calls py line 181 in aresult return await asyncio wrap future self future ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect internal concurrency calls py line 218 in run async result = await coro ^^^^^^^^^^ File home user myproject flows my flow py line 51 in my flow data to store = ^ File home user myproject flows my flow py line 52 in lt listcomp gt item for sublist in data to store futures for item in sublist result ^^^^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect futures py line 231 in result result ^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect internal concurrency calls py line 173 in result return self future result timeout=timeout ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File home obendidi pyenv versions 3 11 2 lib python3 11 concurrent futures base py line 456 in result return self get result ^^^^^^^^^^^^^^^^^^^ File home obendidi pyenv versions 3 11 2 lib python3 11 concurrent futures base py line 401 in get result raise self exception File home user myproject venv lib python3 11 site packages prefect internal concurrency calls py line 218 in run async result = await coro ^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect futures py line 240 in result return await final state result raise on failure=raise on failure fetch=True ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect states py line 91 in get state result raise await get state exception state File home user myproject venv lib python3 11 site packages prefect task runners py line 295 in run and store result self results key = await call ^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect engine py line 1369 in begin task run state = await orchestrate task run ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect engine py line 1494 in orchestrate task run state = await propose state ^^^^^^^^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect engine py line 1930 in propose state response = await set state and handle waits set state ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect engine py line 1917 in set state and handle waits response = await set state func ^^^^^^^^^^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect client orchestration py line 1894 in set task run state response = await ^^^^^^^^^^^^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages httpx client py line 1848 in post return await self request ^^^^^^^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages httpx client py line 1533 in request return await self send request auth=auth follow redirects=follow redirects ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File home user myproject venv lib python3 11 site packages prefect client base py line 269 in send response raise for status File home user myproject venv lib python3 11 site packages prefect client base py line 132 in raise for status raise PrefectHTTPStatusError from httpx error exc from exc cause prefect exceptions PrefectHTTPStatusError Server error 500 Internal Server Error for url Response exception message Internal Server Error For more information check 17 55 28 870 | ERROR | Flow run ebony starfish Finished in state Failed Flow run encountered an exception PrefectHTTPStatusError Server error 500 Internal Server Error for url nResponse exception message Internal Server Error nFor more information check n Traceback most recent call last ```

Zanie · Answer

We ll need the server logs to determine the cause of the 500

Zanie · Answer

We re also adding the ability to add custom retry codes in the next version today so you could retry on 500 we don t by default because it may not be idempotent