Hi when you have a task ORION which returns a large output o

Question

Hi when you have a task ORION which returns a large output or is slow to return it will cause a hiccup with prefect log handlers see chat for details I am using latest beta release 2 03b

Anna Geller · Answer

could you move the code blocks to the thread to keep the main channel cleaner

Anna Geller · Answer

Thanks for reporting that I was able to reproduce the error With smaller numbers up to 100 000 it was fine but going to a million already got my intelligent cow stuck slightly smiling face I checked that already the first task returning a large number causes the issue so it doesn t seem to be related to passing data to downstream tasks I checked that the issue is the same regardless of which task runner you assign as expected I was also able to confirm that the issue here is with logs running the flow with WARNING logs only worked you may use it as a temporary workaround ```PREFECT LOGGING LEVEL=WARNING python community bug py``` < Marvin> open When returning large objects send logs raises an error `RuntimeError Set changed size during iteration`

Marvin · Answer

Malthe Karbo · Answer

Yes Using the sample below ```from prefect import flow task from prefect task runners import SequentialTaskRunner task def slow task big return gt list dict return x i for i in range 10000000 task def fast task uses input input list dict return True flow task runner=SequentialTaskRunner def failing flow results = slow task big return fast task uses input results failing flow ``` will cause ```08 45 56 915 | INFO | prefect engine Created flow run miniature muskox for flow failing flow 08 45 56 916 | INFO | Flow run miniature muskox Using task runner SequentialTaskRunner 08 45 57 069 | WARNING | Flow run miniature muskox No default storage is configured on the server Results from this flow run will be stored in a temporary directory in its runtime environment 08 45 57 488 | INFO | Flow run miniature muskox Created task run slow task big return 36221a7f 0 for task slow task big return Exception in thread orion log worker Traceback most recent call last File usr lib python3 9 threading py line 954 in bootstrap inner self run File usr lib python3 9 threading py line 892 in run self target self args self kwargs File path venv lib python3 9 site packages prefect logging handlers py line 82 in send logs loop anyio run self send logs File path venv lib python3 9 site packages anyio core eventloop py line 56 in run return asynclib run func args backend options File path venv lib python3 9 site packages anyio backends asyncio py line 233 in run return native run wrapper debug=debug File usr lib python3 9 asyncio runners py line 47 in run cancel all tasks loop File usr lib python3 9 asyncio runners py line 56 in cancel all tasks to cancel = tasks all tasks loop File usr lib python3 9 asyncio tasks py line 53 in all tasks tasks = list all tasks File usr lib python3 9 weakrefset py line 61 in iter for itemref in self data RuntimeError Set changed size during iteration``` Moreover it will hang in process forever and if you are on orion cloud it will create a never ending flow run

Anna Geller · Answer

thanks for moving the code block thank you

Malthe Karbo · Answer

And thank you for the fix suggestion will check try it out tomorrow