Johan sh
02/01/2025, 10:51 AMAn error occurred while monitoring flow run 'xxxxxxxxx'. The flow run will not be marked as failed, but an issue may have occurred.
Traceback (most recent call last):
File "/home/azureuser/prefect-env/lib/python3.12/site-packages/prefect/workers/base.py", line 1011, in _submit_run_and_capture_errors
result = await self.run(
^^^^^^^^^^^^^^^
File "/home/azureuser/prefect-env/lib/python3.12/site-packages/prefect_azure/workers/container_instance.py", line 595, in run
status_code = await run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/azureuser/prefect-env/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 235, in run_sync_in_worker_thread
result = await anyio.to_thread.run_sync(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/azureuser/prefect-env/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/azureuser/prefect-env/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2461, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/home/azureuser/prefect-env/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 962, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/azureuser/prefect-env/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 245, in call_with_mark
return call()
^^^^^^
File "/home/azureuser/prefect-env/lib/python3.12/site-packages/prefect_azure/workers/container_instance.py", line 778, in _watch_task_and_get_exit_code
last_log_time = self._get_and_stream_output(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/azureuser/prefect-env/lib/python3.12/site-packages/prefect_azure/workers/container_instance.py", line 871, in _get_and_stream_output
return self._stream_output(logs, last_log_time)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/azureuser/prefect-env/lib/python3.12/site-packages/prefect_azure/workers/container_instance.py", line 948, in _stream_output
if line_time > last_written_time:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: can't compare offset-naive and offset-aware datetimes
Marvin
02/01/2025, 10:51 AMMarvin
02/01/2025, 10:52 AMMarvin
02/01/2025, 10:53 AMJohan sh
02/01/2025, 10:53 AMAn error occurred while monitoring flow run 'xxxxxxxxx'. The flow run will not be marked as failed, but an issue may have occurred.
..........
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/azureuser/prefect-env/lib/python3.12/site-packages/prefect_azure/workers/container_instance.py", line 871, in _get_and_stream_output
return self._stream_output(logs, last_log_time)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/azureuser/prefect-env/lib/python3.12/site-packages/prefect_azure/workers/container_instance.py", line 948, in _stream_output
if line_time > last_written_time:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: can't compare offset-naive and offset-aware datetimes
Marvin
02/01/2025, 10:53 AMTypeError
related to comparing offset-naive and offset-aware datetimes in the Azure Container Instances (ACI) worker for Prefect. This error typically arises when there's a mismatch in timezone awareness between two datetime objects.
Here's what you can do to address this issue:
1. Ensure Datetime Consistency: Make sure that all datetime objects used in your flow are either offset-naive or offset-aware. You can use the pendulum
library to handle timezone-aware datetimes effectively.
2. Review Prefect Azure Configuration: Double-check your ACI work pool configuration to ensure that all settings related to time (e.g., schedules, logs) are consistent and correctly set up. You can refer to the Prefect Azure ACI Worker documentation for guidance.
3. Check for Known Issues: Although no specific GitHub issues were found for this error, it's a good practice to keep your Prefect and Prefect Azure packages updated, as fixes for such issues might be included in newer releases.
If the problem persists, consider reaching out to the Prefect community or support channels with details of your setup and error logs. They might have encountered similar issues and could offer a solution.