https://prefect.io logo
#prefect-community
Title
# prefect-community
b

Blake Stefansen

11/08/2022, 7:27 PM
Hey everyone, I seem to be getting an
Orion logging error
with 2.6.6 for our k8 jobs Will post logs in the thread
Here is the k8 job log
Copy code
--- Orion logging error ---
The log worker encountered a fatal error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/prefect/logging/handlers.py", line 82, in _send_logs_loop
    anyio.run(self.send_logs)
  File "/usr/local/lib/python3.10/dist-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/lib/python3.10/asyncio/runners.py", line 47, in run
    _cancel_all_tasks(loop)
  File "/usr/lib/python3.10/asyncio/runners.py", line 56, in _cancel_all_tasks
    to_cancel = tasks.all_tasks(loop)
  File "/usr/lib/python3.10/asyncio/tasks.py", line 54, in all_tasks
    tasks = list(_all_tasks)
  File "/usr/lib/python3.10/_weakrefset.py", line 65, in __iter__
    for itemref in self.data:
RuntimeError: Set changed size during iteration
Worker information:
    Approximate queue length: 0
    Pending log batch length: 0
    Pending log batch size: 0
I've only seen this once I don't know if this has something to do with our timeout paramater. We are using the prefect python library to create deployment runs
Copy code
from prefect.deployments import run_deployment    

            run_deployment(
                name='geocoder-flow/geocoder_yaml_deploy_k8',
                parameters=geocoder_params,
                flow_run_name=flow_run_name,
                timeout=0
            )
m

Mason Menges

11/11/2022, 10:31 PM
Hey @Blake Stefansen we're tracking related issues for this here https://github.com/PrefectHQ/prefect/issues/7512 if you have some time to run through reporting the behavior you were seeing here
3 Views