09:07:02.450 | DEBUG   | prefect.runner - Runner has no active flow runs or deployments. Sending message to loop service that no further cancellation checks are needed.
09:07:02.450 | DEBUG   | prefect.runner - Checking for cancelled flow runs...
09:07:02.451 | DEBUG   | prefect.utilities.services.critical_service_loop - Run of functools.partial(<bound method Runner._check_for_cancelled_flow_runs of Runner(name='runner-a3d797f4-c5f5-4edd-8099-d0a4be8a217e')>, should_stop=<function Runner.execute_flow_run.<locals>.<lambda> at 0x7f29e00365c0>, on_stop=<bound method CancelScope.cancel of <anyio._backends._asyncio.CancelScope object at 0x7f29df4b5910>>) cancelled
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/utilities/services.py", line 61, in critical_service_loop
    await workload()
  File "/usr/local/lib/python3.11/site-packages/prefect/runner/runner.py", line 737, in _check_for_cancelled_flow_runs
    named_cancelling_flow_runs = await self._client.read_flow_runs(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/orchestration.py", line 2105, in read_flow_runs
    response = await <http://self._client.post|self._client.post>("/flow_runs/filter", json=body)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1892, in post
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1574, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 327, in send
    response = await self._send_with_retry(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 251, in _send_with_retry
    response = await send(request, *send_args, **send_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1661, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1689, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1726, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1763, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 373, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 216, in handle_async_request
    raise exc from None
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 196, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 99, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 74, in handle_async_request
    async with self._request_lock:
  File "/usr/local/lib/python3.11/site-packages/httpcore/_synchronization.py", line 76, in __aenter__
    await self._anyio_lock.acquire()
  File "/usr/local/lib/python3.11/site-packages/anyio/_core/_synchronization.py", line 158, in acquire
    await checkpoint_if_cancelled()
  File "/usr/local/lib/python3.11/site-packages/anyio/lowlevel.py", line 39, in checkpoint_if_cancelled
    await get_async_backend().checkpoint_if_cancelled()
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2065, in checkpoint_if_cancelled
    await sleep(0)
  File "/usr/local/lib/python3.11/asyncio/tasks.py", line 640, in sleep
    await __sleep0()
  File "/usr/local/lib/python3.11/asyncio/tasks.py", line 634, in __sleep0
    yield
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f29df4b5910
09:07:02.456 | DEBUG   | prefect.runner - Stopping runner...

Creating Kubernetes job...
10:06:05 AM
prefect.flow_runs.worker
Worker 'KubernetesWorker 550fcdbc-d976-4802-8df6-a380ad50a22d' submitting flow run '03ef19d0-0601-48b6-9cc1-d0ee87ec5262'
10:06:04 AM
prefect.flow_runs.worker