https://prefect.io logo
Join Slack
Powered by
I have a set of flows that seem to run fine for a ...
# ask-community
m
I have a set of flows that seem to run fine for a while and then the Prefect 3 server crashes saying something about "Cancelled by cancel scope X." I can't predict when it'll happen. This is new behavior that didn't used to happen. Maybe I updated something into an inconsistent state? I'm not really sure where to start debugging. Does anyone have ideas what could cause this or suggestions of where to start looking for the problem?
b
Hey Marcus! This error seems to be somewhat similar to the one present in this issue. From what I gather, this error can be safely ignored and shouldn't be a cause for concern (at least in the context of the other user's report). Does the error always coincide with the server going offline?
m
It does seem to always happen with the server going down.
It looks like right before it crashes I get a flow that does:
Copy code
Encountered exception during execution: PrefectHTTPStatusError("Server error '500 Internal Server Error' for url '<http://127.0.0.1:4200/api/flow_runs/787dc3cb-b274-4932-8ab8-8be4fb88548d/set_state>'\nResponse:

 {'exception_message': 'Internal Server Error'}\nFor more information check: <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500>

")
Traceback (most recent call last):
  File "/home/marcus.hughes/build4/venv/lib64/python3.11/site-packages/prefect/flow_engine.py", line 655, in run_context
    yield self
  File "/home/marcus.hughes/build4/venv/lib64/python3.11/site-packages/prefect/flow_engine.py", line 699, in run_flow_sync
    engine.call_flow_fn()
  File "/home/marcus.hughes/build4/venv/lib64/python3.11/site-packages/prefect/flow_engine.py", line 679, in call_flow_fn
    self.handle_success(result)
  File "/home/marcus.hughes/build4/venv/lib64/python3.11/site-packages/prefect/flow_engine.py", line 282, in handle_success
    self.set_state(terminal_state)
  File "/home/marcus.hughes/build4/venv/lib64/python3.11/site-packages/prefect/flow_engine.py", line 230, in set_state
    state = propose_state_sync(
            ^^^^^^^^^^^^^^^^^^^
  File "/home/marcus.hughes/build4/venv/lib64/python3.11/site-packages/prefect/utilities/engine.py", line 534, in propose_state_sync
    response = set_state_and_handle_waits(set_state)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marcus.hughes/build4/venv/lib64/python3.11/site-packages/prefect/utilities/engine.py", line 518, in set_state_and_handle_waits
    response = set_state_func()
               ^^^^^^^^^^^^^^^^
  File "/home/marcus.hughes/build4/venv/lib64/python3.11/site-packages/prefect/client/orchestration.py", line 3982, in set_flow_run_state
    response = <http://self._client.post|self._client.post>(
               ^^^^^^^^^^^^^^^^^^
  File "/home/marcus.hughes/build4/venv/lib64/python3.11/site-packages/httpx/_client.py", line 1157, in post
    return self.request(
           ^^^^^^^^^^^^^
  File "/home/marcus.hughes/build4/venv/lib64/python3.11/site-packages/httpx/_client.py", line 837, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marcus.hughes/build4/venv/lib64/python3.11/site-packages/prefect/client/base.py", line 575, in send
    response.raise_for_status()
  File "/home/marcus.hughes/build4/venv/lib64/python3.11/site-packages/prefect/client/base.py", line 174, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<http://127.0.0.1:4200/api/flow_runs/787dc3cb-b274-4932-8ab8-8be4fb88548d/set_state>'


Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500>

03:14:22 PM
prefect.flow_runs

Finished in state Failed("Flow run encountered an exception: PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<http://127.0.0.1:4200/api/flow_runs/787dc3cb-b274-4932-8ab8-8be4fb88548d/set_state>'\nResponse:

 {'exception_message': 'Internal Server Error'}\nFor more information check: <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500>
13 Views
Open in Slack
PreviousNext
Powered by