Deepanshu Aggarwal

12/16/2022, 6:21 PM
hi all. i have one question for people running prefect orion on their self hosted instances.. do you guys get a lot of 502 errors? any possible way to avoid these ? adding retries in tasks and flow does help but it would be great if they could be avoided all together. thank you

Christopher Boyd

12/16/2022, 6:27 PM
where are you seeing 502 gateway errors - this is generally a network error specifically where you’re getting an invalid response, either due to network / transient issues, a proxy, resource limitations that the server can’t satisfy

Deepanshu Aggarwal

12/16/2022, 6:40 PM
Encountered exception during execution:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/", line 1346, in orchestrate_task_run
    result = await run_sync(task.fn, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/prefect/utilities/", line 69, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/usr/local/lib/python3.9/site-packages/anyio/", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.9/site-packages/anyio/_backends/", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.9/site-packages/anyio/_backends/", line 867, in run
    result =, *args)
  File "flows/process_sso_integrations_flows/", line 87, in sso_token_executor
    output = run_deployment(
  File "/usr/local/lib/python3.9/site-packages/prefect/utilities/", line 201, in coroutine_wrapper
    return run_async_from_worker_thread(async_fn, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/prefect/utilities/", line 152, in run_async_from_worker_thread
  File "/usr/local/lib/python3.9/site-packages/anyio/", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/usr/local/lib/python3.9/site-packages/anyio/_backends/", line 970, in run_async_from_thread
    return f.result()
  File "/usr/local/lib/python3.9/concurrent/futures/", line 446, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/", line 391, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/prefect/client/", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/prefect/", line 135, in run_deployment
    flow_run = await client.read_flow_run(flow_run_id)
  File "/usr/local/lib/python3.9/site-packages/prefect/client/", line 1444, in read_flow_run
    response = await self._client.get(f"/flow_runs/{flow_run_id}")
  File "/usr/local/lib/python3.9/site-packages/httpx/", line 1757, in get
    return await self.request(
  File "/usr/local/lib/python3.9/site-packages/httpx/", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.9/site-packages/prefect/client/", line 251, in send
  File "/usr/local/lib/python3.9/site-packages/httpx/", line 745, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Server error '502 Bad Gateway' for url '<>'
For more information check: <>
im getting it in the run deployment function
so maybe increasing the timeouts, resources will help ?
ive already set the db timeouts and api timeouts to be around 10 min