https://prefect.io logo
#prefect-community
Title
# prefect-community
d

Daniel Sääf

05/21/2022, 11:45 AM
Hi. I continue to ask my questions here - super grateful for the help i got so far. I have now setup a flow that launches a number of subflows that fetches data from Google Cloud Storage and writes the data to Big Query. I’m running the flow locally (prefect 2.0) and it’s connected to prefect cloud. However, i sometimes run into an error i cannot really understand and i don’t know how to troubleshoot it. It only happens occasionally and i cannot really reproduce it. I moved the trace to the thread. The logging message @
16:34:10.826
is the last thing that happens in the task read_blob in which the error occurs in. So it looks to me that something goes wrong when reporting the task. The error message doens’t tell me that much - so if you have any advices on how i should troubleshoot this i would be really helpful (or if you can guess on what might be wrong?)
a

Anna Geller

05/21/2022, 11:48 AM
can you move the traceback into the thread?
👍 1
it looks like your task run crashes sometimes:
Copy code
Task run 'read_blob-eaa21aac-2' - Crash detected! Execution was interrupted by an unexpected exception.
is it some long running task? is is the task reading the blob object?
could it be that your task that fails is returning some client or other unserializable object? one thing you could do for troubleshooting is to check whether something went wrong when submitting the task to the task runner - the default is ConcurrentTaskRunner - you could do that by moving this failing task directly into the flow. Reason: flows block execution while tasks return immediately a future
z

Zanie

05/21/2022, 6:52 PM
The details of a crashed task run will be included as a debug level log
I’d recommend changing the log level
d

Daniel Sääf

05/21/2022, 8:55 PM
Thank you both for your feedback. Realize that i might revisit how i defined the task - to see if that make things run smoother. And thanks for the advice regarding log level - i will change! If post the stack trace for reference in the thread: I get the following error message:
Copy code
16:34:10.826 | INFO    | Task run 'read_blob-eaa21aac-2' - Fetched z8hobwiqtxc0_2022-05-20T110000_adf3949b6711af3e17e519b166d653ee_c3575d.csv.gz - 57334 rows
16:34:19.665 | INFO    | Task run 'read_blob-eaa21aac-2' - Crash detected! Execution was interrupted by an unexpected exception.
16:34:29.330 | ERROR   | Flow run 'myrtle-millipede' - Encountered exception during execution:
Traceback (most recent call last):
  File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/marketingdata-pZAGOfI1-py3.10/lib/python3.10/site-packages/prefect/engine.py", line 470, in orchestrate_flow_run
    await wait_for_task_runs_and_report_crashes(
  File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/marketingdata-pZAGOfI1-py3.10/lib/python3.10/site-packages/prefect/engine.py", line 882, in wait_for_task_runs_and_report_crashes
    result = await client.set_task_run_state(
  File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/marketingdata-pZAGOfI1-py3.10/lib/python3.10/site-packages/prefect/client.py", line 1688, in set_task_run_state
    response = await <http://self._client.post|self._client.post>(
  File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/marketingdata-pZAGOfI1-py3.10/lib/python3.10/site-packages/httpx/_client.py", line 1820, in post
    return await self.request(
  File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/marketingdata-pZAGOfI1-py3.10/lib/python3.10/site-packages/httpx/_client.py", line 1506, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/marketingdata-pZAGOfI1-py3.10/lib/python3.10/site-packages/prefect/client.py", line 233, in send
    response.raise_for_status()
  File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/marketingdata-pZAGOfI1-py3.10/lib/python3.10/site-packages/httpx/_models.py", line 1510, in raise_for_status    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Server error '500 Internal Server Error' for url '<https://api-beta.prefect.io/api/accounts/df91f491-10a1-4937-abc9-fd13842df41f/workspaces/caa7e1bf-faa2-4ba5-baab-1987efa226f1/task_runs/ad18c70f-1c42-45af-8655-a3143bc874e7/set_state>'
For more information check: <https://httpstatuses.com/500>
16:34:35.466 | ERROR   | Flow run 'myrtle-millipede' - Finished in state Failed('Flow run encountered an exception.')
👍 1
3 Views