https://prefect.io logo
Powered by
Title
s

Sebastian Seitner

08/26/2022, 7:04 AM
Hi, I am occasionally experiencing ‘500 Internal Server Errors’ from the cloud API at (https://api.prefect.cloud/api/accounts/….) while executing a deployed flow in an agent. The error occurs at random points of execution of the flow and leads to a total stop of the flow. I unfortunately have no clue why this happens. I am currently using Prefect 2.1.1 Traceback is in the thread. I would be glad if you could help me.
✅ 1
Encountered exception during execution:
Traceback (most recent call last):
  File "/opt/prefect/environments/psiexport/venv/lib/python3.9/site-packages/prefect/engine.py", line 559, in orchestrate_flow_run
    result = await flow_call()
  File "/tmp/tmpssg5cufqprefect/psiexport/flows/export.py", line 286, in export_round
    open_tasks, metrics = await export_image_data(round, sensors, export_destination, storage_options)
  File "/tmp/tmpssg5cufqprefect/psiexport/flows/export.py", line 253, in export_image_data
    sensor_metrics = await export_images_for_sensor(imagings, settings.TMP_FILE_DESTINATION, target_destination,
  File "/tmp/tmpssg5cufqprefect/psiexport/flows/export.py", line 228, in export_images_for_sensor
    await file_index.materialize()
  File "/tmp/tmpssg5cufqprefect/psiexport/common/file_index.py", line 65, in materialize
    self._add(imaging, round, tray, await files_future.result(), await file_names.result(), base_path)
  File "/opt/prefect/environments/psiexport/venv/lib/python3.9/site-packages/prefect/futures.py", line 227, in _result
    return final_state.result(raise_on_failure=raise_on_failure)
  File "/opt/prefect/environments/psiexport/venv/lib/python3.9/site-packages/prefect/orion/schemas/states.py", line 145, in result
    raise data
  File "/opt/prefect/environments/psiexport/venv/lib/python3.9/site-packages/prefect/task_runners.py", line 316, in _run_and_store_result
    self._results[run_key] = await run_fn(**run_kwargs)
  File "/opt/prefect/environments/psiexport/venv/lib/python3.9/site-packages/prefect/engine.py", line 963, in begin_task_run
    return await orchestrate_task_run(
  File "/opt/prefect/environments/psiexport/venv/lib/python3.9/site-packages/prefect/engine.py", line 1100, in orchestrate_task_run
    state = await client.propose_state(
  File "/opt/prefect/environments/psiexport/venv/lib/python3.9/site-packages/prefect/client.py", line 1800, in propose_state
    response = await self.set_task_run_state(
  File "/opt/prefect/environments/psiexport/venv/lib/python3.9/site-packages/prefect/client.py", line 1883, in set_task_run_state
    response = await <http://self._client.post|self._client.post>(
  File "/opt/prefect/environments/psiexport/venv/lib/python3.9/site-packages/httpx/_client.py", line 1820, in post
    return await self.request(
  File "/opt/prefect/environments/psiexport/venv/lib/python3.9/site-packages/httpx/_client.py", line 1506, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/opt/prefect/environments/psiexport/venv/lib/python3.9/site-packages/prefect/client.py", line 281, in send
    response.raise_for_status()
  File "/opt/prefect/environments/psiexport/venv/lib/python3.9/site-packages/prefect/client.py", line 227, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<https://api.prefect.cloud/api/accounts/32c4c431-62e5-4cf4-a1b9-41f541098e3a/workspaces/8075f69a-2043-4f61-869b-7705e7a9638c/task_runs/79994af4-3418-4322-a785-4373308d7b60/set_state>'
Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://httpstatuses.com/500>
j

Jeff Hale

08/26/2022, 11:25 AM
Thank you for the report. Can you please share the result of 
prefect version
? And if you’re able to share your flow code, that’d be great. Does it happen when running just one specific flow?
s

Sebastian Seitner

08/29/2022, 7:47 AM
The flow is quite big so it is not easy to share and it also contains some internal parts which I can’t share easily. Currently I don’t have a small reproducible sample, because I have no clue what is causing this and to what it could be related. Basically the flow is downloading files from an internal API, processes them, and uploads them to Azure blob storage. So it is mostly network IO. For now I have only experienced it with this flow. But it is more or less the only flow I am currently running regularly. As it only happens ocassionally it is hard to tell if it would also happen in other flows. When it breaks it also does so at different parts within the flow. What I mean by that is that the point the stacktrace leaves my own code is different every time. prefect version: 
Version:             2.1.1
API version:         0.8.0
Python version:      3.9.5
Git commit:          dc2ba222
Built:               Thu, Aug 18, 2022 10:18 AM
OS/Arch:             darwin/x86_64
Profile:             cloud
Server type:         hosted
It happened again from yesterday to today. The similarity I can start to see now is that it seems to always occur on calls to .result() on a PrefectFuture.
j

Jeff Hale

08/30/2022, 11:48 AM
Thank you Sebastian. We expect to have a new release out today. I suggest upgrading to that when it comes out and see if that solves the problem. If it doesn’t, I suggest creating an a bug report issue at https://github.com/PrefectHQ/prefect/issues/new/choose.
👍 1
s

Sebastian Seitner

08/30/2022, 12:57 PM
Thank you for the information. I will test if the new release will fix the issue.
3 Views
#prefect-communityJoin Slack
Powered by