Blake Stefansen

11/08/2022, 7:27 PM
Hey everyone, I seem to be getting an
Orion logging error
with 2.6.6 for our k8 jobs Will post logs in the thread
Here is the k8 job log
--- Orion logging error ---
The log worker encountered a fatal error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/prefect/logging/", line 82, in _send_logs_loop
  File "/usr/local/lib/python3.10/dist-packages/anyio/_core/", line 70, in run
    return, *args, **backend_options)
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/lib/python3.10/asyncio/", line 47, in run
  File "/usr/lib/python3.10/asyncio/", line 56, in _cancel_all_tasks
    to_cancel = tasks.all_tasks(loop)
  File "/usr/lib/python3.10/asyncio/", line 54, in all_tasks
    tasks = list(_all_tasks)
  File "/usr/lib/python3.10/", line 65, in __iter__
    for itemref in
RuntimeError: Set changed size during iteration
Worker information:
    Approximate queue length: 0
    Pending log batch length: 0
    Pending log batch size: 0
I've only seen this once I don't know if this has something to do with our timeout paramater. We are using the prefect python library to create deployment runs
from prefect.deployments import run_deployment    


Mason Menges

11/11/2022, 10:31 PM
Hey @Blake Stefansen we're tracking related issues for this here if you have some time to run through reporting the behavior you were seeing here