Tim-Oliver
05/08/2023, 4:07 PMNate
05/08/2023, 7:51 PMTim-Oliver
05/09/2023, 7:15 AM08:53:10.000 | ERROR | prefect.engine - Engine execution of flow run 'ec3db885-19aa-4e71-997c-f89c39569de4' exited with unexpected exception
Traceback (most recent call last):
File "<frozen importlib._bootstrap_external>", line 786, in exec_module
File "<frozen importlib._bootstrap_external>", line 922, in get_code
File "<frozen importlib._bootstrap_external>", line 979, in get_data
FileNotFoundError: [Errno 2] No such file or directory: 'src/koopaflows/meta_flows/brain_cell_flow_3d.py'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/path/to/envs/koopa/lib/python3.9/site-packages/prefect/engine.py", line 2158, in <module>
enter_flow_run_engine_from_subprocess(flow_run_id)
File "/path/to/envs/koopa/lib/python3.9/site-packages/prefect/engine.py", line 202, in enter_flow_run_engine_from_subprocess
return from_sync.wait_for_call_in_loop_thread(
File "/path/to/envs/koopa/lib/python3.9/site-packages/prefect/client/schemas.py", line 107, in result
return get_state_result(self, raise_on_failure=raise_on_failure, fetch=fetch)
File "/path/to/envs/koopa/lib/python3.9/site-packages/prefect/states.py", line 76, in get_state_result
return _get_state_result(state, raise_on_failure=raise_on_failure)
File "/path/to/envs/koopa/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 260, in coroutine_wrapper
return call()
File "/path/to/envs/koopa/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 245, in __call__
return self.result()
File "/path/to/envs/koopa/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 173, in result
return self.future.result(timeout=timeout)
File "/path/to/envs/koopa/lib/python3.9/concurrent/futures/_base.py", line 433, in result
return self.__get_result()
File "/path/to/envs/koopa/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/path/to/envs/koopa/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 218, in _run_async
result = await coro
File "/path/to/envs/koopa/lib/python3.9/site-packages/prefect/states.py", line 91, in _get_state_result
raise await get_state_exception(state)
File "/path/to/envs/koopa/lib/python3.9/site-packages/prefect/engine.py", line 276, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "/path/to/envs/koopa/lib/python3.9/site-packages/prefect/client/utilities.py", line 40, in with_injected_client
return await fn(*args, **kwargs)
File "/path/to/envs/koopa/lib/python3.9/site-packages/prefect/deployments.py", line 217, in load_flow_from_flow_run
flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
File "/path/to/envs/koopa/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(
File "/path/to/envs/koopa/lib/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/path/to/envs/koopa/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/path/to/envs/koopa/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/path/to/envs/koopa/lib/python3.9/site-packages/prefect/flows.py", line 809, in load_flow_from_entrypoint
flow = import_object(entrypoint)
File "/path/to/envs/koopa/lib/python3.9/site-packages/prefect/utilities/importtools.py", line 201, in import_object
module = load_script_as_module(script_path)
File "/path/to/envs/koopa/lib/python3.9/site-packages/prefect/utilities/importtools.py", line 164, in load_script_as_module
raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'src/koopaflows/meta_flows/brain_cell_flow_3d.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
srun: error: vcl1049: task 0: Exited with exit code 1
08:53:10.499 | ERROR | prefect.infrastructure.process - Process 'koopa' exited with status code: 1
Somehow the github repos is not cloned. I can clone the repo myself and copy the content into the working directory, then it works.Tim-Oliver
05/09/2023, 7:16 AM08:52:53.517 | ERROR | prefect._internal.concurrency.services - Service 'EventsWorker' failed with 4 pending items.
Traceback (most recent call last):
File "/path/to/envs/prefect/lib/python3.9/site-packages/prefect/_internal/concurrency/services.py", line 119, in _run
async with self._lifespan():
File "/path/to/envs/prefect/lib/python3.9/contextlib.py", line 181, in __aenter__
return await self.gen.__anext__()
File "/path/to/envs/prefect/lib/python3.9/site-packages/prefect/events/worker.py", line 30, in _lifespan
async with self._client:
File "/path/to/envs/prefect/lib/python3.9/site-packages/prefect/events/clients.py", line 118, in __aenter__
await self._reconnect()
File "/path/to/envs/prefect/lib/python3.9/site-packages/prefect/events/clients.py", line 136, in _reconnect
self._websocket = await self._connect.__aenter__()
File "/path/to/envs/prefect/lib/python3.9/site-packages/websockets/legacy/client.py", line 642, in __aenter__
return await self
File "/path/to/envs/prefect/lib/python3.9/site-packages/websockets/legacy/client.py", line 659, in __await_impl_timeout__
return await asyncio.wait_for(self.__await_impl__(), self.open_timeout)
File "/path/to/envs/prefect/lib/python3.9/asyncio/tasks.py", line 479, in wait_for
return fut.result()
File "/path/to/envs/prefect/lib/python3.9/site-packages/websockets/legacy/client.py", line 663, in __await_impl__
_transport, _protocol = await self._create_connection()
File "/path/to/envs/prefect/lib/python3.9/asyncio/base_events.py", line 1073, in create_connection
raise OSError('Multiple exceptions: {}'.format(
OSError: Multiple exceptions: [Errno 111] Connect call failed ('34.111.36.90', 443), [Errno 101] Network is unreachable
Tim-Oliver
05/09/2023, 7:33 AMTim-Oliver
05/09/2023, 8:02 AMDownloading flow code from storage at ''
Nate
05/09/2023, 2:17 PMprefect version
output? will try to reproduceTim-Oliver
05/09/2023, 2:29 PMNate
05/09/2023, 7:32 PMwould you be able to upgrade your agent to the newest prefect version?Copy codeService 'EventsWorker' failed with 4 pending items.
Tim-Oliver
05/10/2023, 7:34 AMConnect call failed ('34.11.36.90', 443)
error. Where does it try to connect to and why?Tim-Oliver
05/10/2023, 7:39 AM