t

    Tomer Cagan

    6 months ago
    I am trying to follow the intro to sub-flow results passing and for some reason, the get_task_run_results task get stack. I am using prefect server and have kubernetes agent (though I am not sure that matter since I use the --execute cli arg.
    Anna Geller

    Anna Geller

    6 months ago
    You can find one example and more info about it here. If this doesn’t help you and you’re still stuck, can you share your flow so that we can check together what’s wrong in your setup?
    t

    Tomer Cagan

    6 months ago
    I see in the log:
    (venv) tomercagan@Tomers-MacBook-Pro prefect-play % prefect run --name "parent" --execute
    Looking up flow metadata... Done
    Creating run for flow 'parent'... Done
    └── Name: classy-piculet
    └── UUID: 10ea2885-93c6-4ca9-9053-9770ed0b4ae0
    └── Labels: ['agentless-run-b8b53ae8']
    └── Parameters: {}
    └── Context: {}
    └── URL: <http://localhost:8080/default/flow-run/10ea2885-93c6-4ca9-9053-9770ed0b4ae0>
    Executing flow run...
    └── 15:18:10 | INFO    | Creating subprocess to execute flow run...
    └── 15:18:10 | INFO    | Beginning Flow run for 'parent'
    └── 15:18:11 | INFO    | Task 'create_flow_run': Starting task run...
    └── 15:18:11 | INFO    | Creating flow run 'classy-piculet-child' for flow 'child'...
    └── 15:18:11 | INFO    | Created flow run 'classy-piculet-child': <http://localhost:8080/default/flow-run/595921bf-7b99-406f-b71a-89e99c35797b>
    └── 15:18:11 | INFO    | Task 'create_flow_run': Finished task run for task with final state: 'Success'
    └── 15:18:11 | INFO    | Task 'get_task_run_result': Starting task run...
    and it gets stack there for a while. Eventually I am getting this:
    └── 14:52:21 | ERROR   | Task 'get_task_run_result': Exception encountered during task execution!
    Traceback (most recent call last):
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
        httplib_response = self._make_request(
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 449, in _make_request
        six.raise_from(e, None)
      File "<string>", line 3, in raise_from
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 444, in _make_request
        httplib_response = conn.getresponse()
      File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1377, in getresponse
        response.begin()
      File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 320, in begin
        version, status, reason = self._read_status()
      File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 281, in _read_status
        line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
      File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 704, in readinto
        return self._sock.recv_into(b)
    ConnectionResetError: [Errno 54] Connection reset by peer
    
    During handling of the above exception, another exception occurred:
    Traceback (most recent call last):
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
        resp = conn.urlopen(
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen
        retries = retries.increment(
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/urllib3/util/retry.py", line 550, in increment
        raise six.reraise(type(error), error, _stacktrace)
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/urllib3/packages/six.py", line 769, in reraise
        raise value.with_traceback(tb)
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
        httplib_response = self._make_request(
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 449, in _make_request
        six.raise_from(e, None)
      File "<string>", line 3, in raise_from
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 444, in _make_request
        httplib_response = conn.getresponse()
      File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1377, in getresponse
        response.begin()
      File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 320, in begin
        version, status, reason = self._read_status()
      File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 281, in _read_status
        line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
      File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 704, in readinto
        return self._sock.recv_into(b)
    urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 876, in get_task_run_state
        value = prefect.utilities.executors.run_task_with_timeout(
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
        return task.run(*args, **kwargs)  # type: ignore
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/prefect/tasks/prefect/flow_run.py", line 223, in get_task_run_result
        flow_run = flow_run.get_latest()
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/prefect/backend/flow_run.py", line 414, in get_latest
        return self.from_flow_run_id(
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/prefect/backend/flow_run.py", line 566, in from_flow_run_id
        flow_run_data = cls._query_for_flow_run(where={"id": {"_eq": flow_run_id}})
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/prefect/backend/flow_run.py", line 608, in _query_for_flow_run
        result = client.graphql(flow_run_query)
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/prefect/client/client.py", line 452, in graphql
        result = <http://self.post|self.post>(
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/prefect/client/client.py", line 407, in post
        response = self._request(
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/prefect/client/client.py", line 641, in _request
        response = self._send_request(
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/prefect/client/client.py", line 506, in _send_request
        response = <http://session.post|session.post>(
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/requests/sessions.py", line 577, in post
        return self.request('POST', url, data=data, json=json, **kwargs)
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
        resp = self.send(prep, **send_kwargs)
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
        r = adapter.send(request, **kwargs)
      File "/Users/tomercagan/dev/venv/lib/python3.9/site-packages/requests/adapters.py", line 501, in send
        raise ConnectionError(err, request=request)
    requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))
    └── 14:52:24 | ERROR   | Failed to set task state with error: ConnectionError(ProtocolError('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer')))
    And this continues on and on
    I used the example flow verbatim from the link above. I will review what you shared.
    Anna Geller

    Anna Geller

    6 months ago
    this error suggests some API or networking issue. Your server need to use a Prefect version which is >= Prefect version used by your agents and flow registration environment
    t

    Tomer Cagan

    6 months ago
    I tried with prefect version 0.15.13 and version 2022.01.25 of the helm chart. I then removed everything, reinstalled the chart and reinstalled prefect to version 1.0.0 (I saw it was just announced 🙂 )
    Anna Geller

    Anna Geller

    6 months ago
    Based on your introduction, it looks you are just getting started with Prefect. We recommend using Prefect Cloud at first, since it’s much easier than using Server. Server adds a lot of additional complexity that you need to manage and Cloud is free to use for 20,000 task runs each month and you don’t even need to give your credit card details when signing up for the free Starter or Standard account. You can always deploy your own Server later, but to get started, I’d recommend using Cloud at first.
    I think it’s quite useful to follow this orchestration layer getting started guide before running any flows, since it will give you an idea of how all components work together https://docs.prefect.io/orchestration/getting-started/quick-start.html
    t

    Tomer Cagan

    6 months ago
    Thanks - I will try cloud and will see I didn't miss anything in the getting started guide
    I signed in for cloud and tried to run it, experiencing the same issue. Only after working through the get started guide and starting an agent I was able to get things to work. I assume the issue is calling the run command with
    --execute
    without first running a local agent.
    Anna Geller

    Anna Geller

    6 months ago
    If you use --execute, you are triggering the so called "agentless" flow run. Check out this excellent blog post, it explains it really well (it still totally applies in Prefect 1.0, not just in 0.15.0)
    t

    Tomer Cagan

    6 months ago
    I read it. I am not sure how this applies to the scenario of having a child flow - I was inspecting it and didn't find the labels I was expecting. I would expect that if I am starting an agentless execution, I will not need any agent but I am not sure this is the case (is there a bug when running parent-child flows with agentless exec?!)
    Anna Geller

    Anna Geller

    6 months ago
    Agentless execution is for a single ad-hoc flow run, so it won't work for running subflows. For that, you need to have an agent running and the proper agent label set on your flow's run config.
    I only shared the blog post to give more background on the --execute flag and when/how to use it
    t

    Tomer Cagan

    6 months ago
    I was trying to use agentless to see how it works. It is worth noting that this is not possible in relevant docs 🙂
    Anna Geller

    Anna Geller

    6 months ago
    Yeah I see it mentioned here https://docs.prefect.io/orchestration/faq/upgrading_1.0.html#running-flows and here https://docs.prefect.io/orchestration/flow-runs/creation.html#cli - if you feel like we can mention it somewhere else, we are happy to accept a docs PR 🙏
    t

    Tomer Cagan

    6 months ago
    In the former the
    --execute
    is described, and in the latter, there is a box talking about agentless running ("Agentless flow run execution") - maybe there would be a good place to mention that calling a sub-flow with local execution would not work?
    Anna Geller

    Anna Geller

    6 months ago
    I see, that's a good point, we can definitely add it as a comment - happy to review a PR if you would want to add it to the docs otherwise let me open an issue for now
    @Marvin open "Extend the docs that agentless execution can only trigger a single flow run with no child flow runs"
    Marvin

    Marvin

    6 months ago
    t

    Tomer Cagan

    6 months ago
    That's a cool feature!