https://prefect.io logo
Title
p

Paco Ibañez

03/16/2023, 7:43 PM
Hello! I have a flow that runs 11 subflows and for some reason the 10th flow always raises a timeout error when calling run_deployment. Any ideas? thank you!
19:10:48.889 | DEBUG   | prefect.client - Connecting to API at <https://acmt-prefect.dip/api/>
19:11:17.655 | DEBUG   | prefect.client - Encountered retryable exception during request. Another attempt will be made in 2s. This is attempt 1/6.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 34, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "/usr/local/lib/python3.8/site-packages/anyio/streams/tls.py", line 195, in receive
    data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
  File "/usr/local/lib/python3.8/site-packages/anyio/streams/tls.py", line 137, in _call_sslobject_method
    data = await self.transport_stream.receive()
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 1265, in receive
    await self._protocol.read_event.wait()
  File "/usr/local/lib/python3.8/asyncio/locks.py", line 309, in wait
    await fut
asyncio.exceptions.CancelledError
...
Traceback (most recent call last):
  File "/src/pipelines/orchestrator/core/orchestrator.py", line 78, in run
    result = self.strategy.execute(pipeline_step_exec_config)
  File "/src/pipelines/orchestrator/core/run_strategy/__init__.py", line 100, in execute
    run = run_deployment(
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 226, in coroutine_wrapper
    return run_async_from_worker_thread(async_fn, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 177, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/usr/local/lib/python3.8/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.8/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/deployments.py", line 123, in run_deployment
    flow_run = await client.create_flow_run_from_deployment(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/orchestration.py", line 481, in create_flow_run_from_deployment
    response = await <http://self._client.post|self._client.post>(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.8/site-packages/prefect/client/base.py", line 229, in send
    response = await self._send_with_retry(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/base.py", line 187, in _send_with_retry
    response = await request()
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectTimeout
I also noticed the log msg `
Connecting to API at <https://acmt-prefect.dev/api/>
that url is defined in the server as
PREFECT_ORION_UI_API_URL
, why does it connect to that instead of what I defined in the agent?
- name: PREFECT_API_URL
              value: <http://prefect-orion:4200/api>