https://prefect.io logo
Channels
Powered by
j

Joseph

02/04/2021, 6:27 PM
I am running a very simple flow with a custom executor (I have an executor that is similar to Dask for a very large private compute cluster). When I run the flow locally (
flow.run()
) it successfully completes and using diagnostic tools for my cluster I can see the jobs complete. I can also register the flow with the server (
flow.register
). But when I quick run the flow, from the UI’s perspective it looks as if it never completes (it is stuck in the “running” state). Using my cluster diagnostics I can see the actual jobs get scheduled and complete. So I’d guess whatever sort of events are meant to be sent to the server in order to indicate state changes in the jobs are not being sent? How can I diagnose what’s happening?
Should I instead implement an agent that inherits from LocalExecutor, i.e. DaskAgent, rather than specifying a custom executor?
Or is that just so that the flow itself, in addition to the tasks, are run in Dask?
Ok, I ran the agent with -f which gives good diagnostics
Copy code
Failed to set task state with error: ConnectionError(MaxRetryError("HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError(
'<urllib3.connection.HTTPConnection object at 0x5619a00a7b50>: Failed to establish a new connection: [Errno 111] Connection refused'))"))
Traceback (most recent call last):
Looks like I need to somehow pass the server url to the jobs that are sent out.
I see this is what DaskAgent does
👀 1
s

Sagun Garg

02/06/2021, 11:12 AM
@Joseph Sorry for jumping in between your thread. Recently I also started getting this error when I am trying this command
Copy code
prefect backend server && prefect server create-tenant --name default --slug default
Below is the error that I am getting wish to understand what is the way out for this issue.
Copy code
(.venv) sagungargs@Saguns-MacBook-Pro flows % prefect backend server && prefect server create-tenant --name default --slug default             
Backend switched to server
Traceback (most recent call last):
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/urllib3/connection.py", line 170, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

.........

  File "/Users/sagungargs/Work/portcast2/flows/.venv/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x105cc33d0>: Failed to establish a new connection: [Errno 61] Connection refused'))
Powered by