https://prefect.io logo
Join Slack
Powered by
Hey folks, I just had a flow run failing because o...
# prefect-community
a
Hey folks, I just had a flow run failing because of "prefect.exceptions.MissingResult: State data is missing. Typically, this occurs when result persistence is disabled and the state has been retrieved from the API." This flow runs several times per day and always succeeded before and also after this incident. The good thing for me: The flow entered failed state and als the logs show an exception - but the flow itself continued after the exception and all the "results" of the pipeline were achieved. Looking at the Exception (in the Thread), is there something I'm missing, or is this maybe something for a github issue? (prefect 2.6.6, prefect cloud)
Copy code
Created task run 'schedule_report-060bbc2f-0' for task 'schedule_report'
11:32:15 PM
Executing 'schedule_report-060bbc2f-0' immediately...
11:32:15 PM
Waiting for report do be finished...
11:32:16 PM
Created task run 'schedule_report-060bbc2f-0' for task 'schedule_report'
11:32:21 PM
Executing 'schedule_report-060bbc2f-0' immediately...
11:32:21 PM
Task run '700c990a-1e10-4f5d-9bac-8c5cdc51b428' already finished.
11:32:21 PM
schedule_report-060bbc2f-0
Encountered exception during execution:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 610, in orchestrate_flow_run
    result = await run_sync(flow_call)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 126, in run_sync_in_interruptible_worker_thread
    async with anyio.create_task_group() as tg:
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 108, in capture_worker_thread_and_result
    result = __fn(*args, **kwargs)
  File "/tmp/tmpc5xrd35cprefect/piano_flows.py", line 98, in integrate_piano_subscription_logs
    report_url = schedule_report(baseurl, aid, piano_token, days_to_sync, logger)
  File "/usr/local/lib/python3.10/site-packages/prefect/tasks.py", line 353, in __call__
    return enter_task_run_engine(
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 733, in enter_task_run_engine
    return run_async_from_worker_thread(begin_run)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 148, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/usr/local/lib/python3.10/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 874, in get_task_call_return_value
    return await future._result()
  File "/usr/local/lib/python3.10/site-packages/prefect/futures.py", line 237, in _result
    return await final_state.result(raise_on_failure=raise_on_failure, fetch=True)
  File "/usr/local/lib/python3.10/site-packages/prefect/states.py", line 86, in _get_state_result
    raise MissingResult(
prefect.exceptions.MissingResult: State data is missing. Typically, this occurs when result persistence is disabled and the state has been retrieved from the API.
11:32:21 PM
Created task run 'download_and_send_report-a95b1685-0' for task 'download_and_send_report'
11:32:23 PM
Executing 'download_and_send_report-a95b1685-0' immediately...
11:32:23 PM
Inserting to Bigquery # rows: 6040
11:32:25 PM
t
I saw this in some of my flows recently. I was running using a 
DaslTaslExecutor
backed by 
dask_jobqueue.SLURMCluster
to start dask works after acquiring SLURM resources. I can not say for certain I know what the error was. But, I noticed in some of my slurm logs that the odd dask worker would pause itself as it say memory consumption got too high. I also saw in my prefect logs a set of tasks entering some weird "Taask XXXXXX has already been run successfully" type message. I reconfigured my resource request in the SLURMCluster and that removed the error.
Of course, none of what I said could be related to your actual use case and problem
m
Looks like another one where #7512 and #7472 might be related indeed.
41 Views
Open in Slack
PreviousNext
Powered by