Andreas Nigg
11/18/2022, 7:27 AMCreated task run 'schedule_report-060bbc2f-0' for task 'schedule_report'
11:32:15 PM
Executing 'schedule_report-060bbc2f-0' immediately...
11:32:15 PM
Waiting for report do be finished...
11:32:16 PM
Created task run 'schedule_report-060bbc2f-0' for task 'schedule_report'
11:32:21 PM
Executing 'schedule_report-060bbc2f-0' immediately...
11:32:21 PM
Task run '700c990a-1e10-4f5d-9bac-8c5cdc51b428' already finished.
11:32:21 PM
schedule_report-060bbc2f-0
Encountered exception during execution:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 610, in orchestrate_flow_run
result = await run_sync(flow_call)
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 126, in run_sync_in_interruptible_worker_thread
async with anyio.create_task_group() as tg:
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
raise exceptions[0]
File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 108, in capture_worker_thread_and_result
result = __fn(*args, **kwargs)
File "/tmp/tmpc5xrd35cprefect/piano_flows.py", line 98, in integrate_piano_subscription_logs
report_url = schedule_report(baseurl, aid, piano_token, days_to_sync, logger)
File "/usr/local/lib/python3.10/site-packages/prefect/tasks.py", line 353, in __call__
return enter_task_run_engine(
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 733, in enter_task_run_engine
return run_async_from_worker_thread(begin_run)
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 148, in run_async_from_worker_thread
return anyio.from_thread.run(call)
File "/usr/local/lib/python3.10/site-packages/anyio/from_thread.py", line 49, in run
return asynclib.run_async_from_thread(func, *args)
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
return f.result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 874, in get_task_call_return_value
return await future._result()
File "/usr/local/lib/python3.10/site-packages/prefect/futures.py", line 237, in _result
return await final_state.result(raise_on_failure=raise_on_failure, fetch=True)
File "/usr/local/lib/python3.10/site-packages/prefect/states.py", line 86, in _get_state_result
raise MissingResult(
prefect.exceptions.MissingResult: State data is missing. Typically, this occurs when result persistence is disabled and the state has been retrieved from the API.
11:32:21 PM
Created task run 'download_and_send_report-a95b1685-0' for task 'download_and_send_report'
11:32:23 PM
Executing 'download_and_send_report-a95b1685-0' immediately...
11:32:23 PM
Inserting to Bigquery # rows: 6040
11:32:25 PM
Tim Galvin
11/18/2022, 7:40 AMDaslTaslExecutor
backed by dask_jobqueue.SLURMCluster
to start dask works after acquiring SLURM resources.
I can not say for certain I know what the error was. But, I noticed in some of my slurm logs that the odd dask worker would pause itself as it say memory consumption got too high. I also saw in my prefect logs a set of tasks entering some weird "Taask XXXXXX has already been run successfully" type message.
I reconfigured my resource request in the SLURMCluster and that removed the error.