When I use map to submit about 10000 tasks the run failed Is

Question

When I use map to submit about 10000 tasks the run failed Is there a limit to how many tasks I can submit

Kha Nguyen · Answer

```Encountered exception during execution Traceback most recent call last File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages prefect engine py line 557 in orchestrate flow run result = await run sync flow call File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages prefect utilities asyncutils py line 56 in run sync in worker thread return await anyio to thread run sync call cancellable=True File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages anyio backends asyncio py line 867 in run result = context run func args File Users kha Desktop demo singularity flow py line 23 in run forecast results = produce forecast map time series horizon=unmapped 14 File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages prefect tasks py line 627 in map return enter task run engine File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages prefect engine py line 686 in enter task run engine return run async from worker thread begin run File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages prefect utilities asyncutils py line 136 in run async from worker thread return anyio from thread run call File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages anyio from thread py line 49 in run return asynclib run async from thread func args File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages anyio backends asyncio py line 970 in run async from thread return f result File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 concurrent futures base py line 446 in result return self get result File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 concurrent futures base py line 391 in get result raise self exception File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages prefect engine py line 737 in begin task map return await gather task runs File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages prefect utilities asyncutils py line 339 in gather async with create gather task group as tg File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages prefect utilities asyncutils py line 317 in aexit retval = await self task group aexit tb File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages anyio backends asyncio py line 662 in aexit raise exceptions 0 File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages prefect utilities asyncutils py line 286 in run and store self results key = await fn args File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages prefect engine py line 783 in create task run then submit task run = await create task run File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages prefect engine py line 823 in create task run task run = await flow run context client create task run File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages prefect client py line 1622 in create task run response = await File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages httpx client py line 1842 in post return await self request File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages httpx client py line 1527 in request return await self send request auth=auth follow redirects=follow redirects File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages prefect client py line 278 in send response raise for status File opt homebrew Caskroom miniforge base envs singularity2 lib python3 10 site packages prefect client py line 224 in raise for status raise PrefectHTTPStatusError from httpx error exc from exc cause prefect exceptions PrefectHTTPStatusError Server error 500 Internal Server Error for url Response exception message Internal Server Error For more information check ```

Kha Nguyen · Answer

This seems to be an issue with httpx

Bianca Hoch · Answer

Hi Kha thanks for reporting Can you describe what your setup looks like a bit more What your agents storage look like etc

Kha Nguyen · Answer

Hi Bianca here is a reproducible example ```from prefect import flow get run logger task task def some task i int gt int logger = get run logger f Doubling i to i 2 return i 2 task def generate numbers g int gt list int return list range g flow def demo flow logger = get run logger ls = generate numbers submit 100 results future = some task map ls results = f wait result for f in results future results demo flow ``` You can replace the argument to `generate numbers` from 100 to 10 and the flow can run successfully However even 100 is enough to break this flow In my case the number can go up to 30000

Kha Nguyen · Answer

Using the python for loop works though ```results future = some task submit i for i in ls wait result ``` So I can say that Prefect has no problem handling a large number of tasks but the httpx config when using the map API has a bug

Kha Nguyen · Answer

< Bianca Hoch> in this example you can also see that I have logging in task but I cannot see that message anywhere

Bianca Hoch · Answer

Hello Kha interestingly enough I was able to have the flow run to completion setting argument for `generate numbers` to 100 200 and even 500

Bianca Hoch · Answer

I was also able to see the log info

Kha Nguyen · Answer

Yes I can see the log with a local process runner too

Kha Nguyen · Answer

I am using Macbook M1 I wonder if that affects anything

Kha Nguyen · Answer

Thanks for reproducing the run

Bianca Hoch · Answer

Thank you for providing something I could reproduce For context I m using a Macbook MacOS 12 3 1 Chip Apple M1 Pro

Bianca Hoch · Answer

Note that if you re still running in to the same problem feel free to open up a GitHub issue like I mentioned in our other exchange

Kha Nguyen · Answer

Hi Bianca I reran my example from yesterday and today it ran successfully without any problem I don t know what causes the exception in the first place Perhaps it was just some local system issue

Bianca Hoch · Answer

Glad to hear that Kha