• r

    Roey Brecher

    1 year ago
    Hello. I’d like to raise the issue of changes in recent prefect versions, that break our deployment. It seems that if Prefect core + server + docker image versions aren’t synchronised to the exact patch version (e.g 0.13.13 might not work with 0.13.17), things might break down. Usually “patch” versions should not include any breaking change. In order for us to make sure our production flows won’t break, we have to make sure that we use specific version across the board. fetching the latest docker image or the latest prefect core version is not something one should do in production, unless one makes sure to also update his Prefect Server (which might not be possible or required in production environments. attached is the latest error we received out of the blue, but this is not the first time we experience a breaking change.
    Unexpected error: ClientError('400 Client Error: Bad Request for url: <http://10.0.4.45:4200/graphql>\n\nThis is likely caused by a poorly formatted GraphQL query or mutation. GraphQL sent:\n\nquery {\n    mutation($input: create_task_run_artifact_input!) {\n            create_task_run_artifact(input: $input) {\n                id\n        }\n    }\n}\nvariables {\n    {"input": {"task_run_id": "5b4f7865-b192-462e-a9ee-753e243db30a", "kind": "link", "data": {"link": "/default/flow-run/4fc7c743-787f-49f7-b3dc-31cfc2fa720f"}, "tenant_id": null}}\n}')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/prefect/engine/runner.py", line 48, in inner
        new_state = method(self, state, *args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 856, in get_task_run_state
        value = prefect.utilities.executors.run_task_with_timeout(
      File "/usr/local/lib/python3.8/site-packages/prefect/utilities/executors.py", line 298, in run_task_with_timeout
        return task.run(*args, **kwargs)  # type: ignore
      File "/usr/local/lib/python3.8/site-packages/prefect/utilities/tasks.py", line 449, in method
        return run_method(self, *args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/prefect/tasks/prefect/flow_run.py", line 173, in run
        create_link(urlparse(run_link).path)
      File "/usr/local/lib/python3.8/site-packages/prefect/artifacts.py", line 31, in create_link
        return client.create_task_run_artifact(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1689, in create_task_run_artifact
        result = self.graphql(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 296, in graphql
        result = <http://self.post|self.post>(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 211, in post
        response = self._request(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 433, in _request
        response = self._send_request(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 365, in _send_request
        raise ClientError(msg)
    prefect.utilities.exceptions.ClientError: 400 Client Error: Bad Request for url: <http://10.0.4.45:4200/graphql>
    
    This is likely caused by a poorly formatted GraphQL query or mutation. GraphQL sent:
    
    query {
        mutation($input: create_task_run_artifact_input!) {
                create_task_run_artifact(input: $input) {
                    id
            }
        }
    }
    variables {
        {"input": {"task_run_id": "5b4f7865-b192-462e-a9ee-753e243db30a", "kind": "link", "data": {"link": "/default/flow-run/4fc7c743-787f-49f7-b3dc-31cfc2fa720f"}, "tenant_id": null}}
    }
    r
    Jim Crist-Harif
    4 replies
    Copy to Clipboard
  • ale

    ale

    1 year ago
    Hi folks, is it possible to update the description of an existing project in Prefect Server using Prefect Client. I see there’s a
    create_project
    API, but nothing related to update
    ale
    Jim Crist-Harif
    +1
    6 replies
    Copy to Clipboard
  • Mike Marinaccio

    Mike Marinaccio

    1 year ago
    Hi All - quick question on restarting failed flows. I’m attempting to restart a flow by 1) setting all tasks to
    Scheduled
    via the UI and 2) rerunning the entire flow. However, the first step fails with the attached UI error. I see no errors in the Chrome console, nor anything in the service logs. This only appears to happen when I select the
    Change the state of all task runs too
    box. I’m running Prefect Server with the latest containers. Any tips? Thanks!
    Mike Marinaccio
    Jenny
    7 replies
    Copy to Clipboard
  • Innokenty Tretyakov

    Innokenty Tretyakov

    1 year ago
    Hi! I wonder how come that BigQueryTask have no ability to return
    query_job
    object. I mean, if one have an idea to log the results of the query (status, proceeded_bytes, etc.), he or she can not get an object to fetch that info. I mean something like that (tasks/gcp/bigquery.py; class BigQueryTask; def run):
  • v

    Vipul

    1 year ago
    Hi All, quick question, is there any quick way to remove all the metadata and log information from Postgres db after 30 days? Thanks
    v
    Jeremiah
    4 replies
    Copy to Clipboard
  • r

    Roey Brecher

    1 year ago
    did anyone encounter issues with Fargate tasks (flows) loosing connection to Prefect Server after a while (running for a few hours)? this is the error we’re seeing.
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
        resp = conn.urlopen(
      File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
        retries = retries.increment(
      File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 446, in increment
        raise MaxRetryError(_pool, url, error or ResponseError(cause))
    urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.0.4.45', port=4200): Max retries exceeded with url: /graphql (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f0de03bb820>, 'Connection to 10.0.4.45 timed out. (connect timeout=30)'))
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/flow_runner.py", line 179, in interrupt_if_cancelling
        flow_run_info = self.client.get_flow_run_info(flow_run_id)
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1015, in get_flow_run_info
        result = self.graphql(query).data.flow_run_by_pk  # type: ignore
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 275, in graphql
        result = <http://self.post|self.post>(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 230, in post
        response = self._request(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 400, in _request
        response = self._send_request(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 319, in _send_request
        response = <http://session.post|session.post>(url, headers=headers, json=params, timeout=30)
      File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 578, in post
        return self.request('POST', url, data=data, json=json, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
        resp = self.send(prep, **send_kwargs)
      File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
        r = adapter.send(request, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 504, in send
        raise ConnectTimeout(e, request=request)
    requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='10.0.4.45', port=4200): Max retries exceeded with url: /graphql (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f0de03bb820>, 'Connection to 10.0.4.45 timed out. (connect timeout=30)'))
  • v

    Vladimir Zoubritsky

    1 year ago
    Hello! I've been trying out the Kubernetes Agent configuration and noticed it was getting killed after some time with failing health checks I've noticed the Agent is listening on
    localhost
    , so it wouldn't be reachable on it's network interface - and changing it to 0.0.0.0 makes health checks pass, are there any security implications for this or it's a good default?https://github.com/PrefectHQ/server/blob/a3626c81f280bf749394c2d00d15d6450b724a44/helm/prefect-server/templates/agent/deployment.yaml#L65
    v
    Michael Adkins
    2 replies
    Copy to Clipboard
  • JC Garcia

    JC Garcia

    1 year ago
    Hello! Let's say I wanted to run multiple dbt commands through the dbt task, but will not know the commands until runtime (they will be received as a paramter). What is the recommended way of looping tasks? Would something like this be ok?
    dbt_task = DbtShellTask(
      ...
    )
    
    with Flow("dbt-flow") as flow:
    
        dbt_command_string = Parameter('dbt_command_string', default = 'dbt run', required = True)
    
        dbt_commands = dbt_command_string.split(',')
    
    
        for command in dbt_commands:
          dbt_task(
              command=dbt_command,
              ...
          )
    JC Garcia
    Michael Adkins
    +1
    9 replies
    Copy to Clipboard
  • JC Garcia

    JC Garcia

    1 year ago
    Another question, is it possible to have prefect's cli run to exit with an error if any of the tasks error out while watching the state or streaming the logs (-w, -l)?
    JC Garcia
    Jeremiah
    3 replies
    Copy to Clipboard
  • p

    Philip Evans

    1 year ago
    Hi there. I am wanting to run a flow on prefect server with a custom schedule. The schedule is set with an interval clock that runs everyday and generates a clock for roughly 4000 parameters.
    schedule = Schedule(
        clocks=[
            IntervalClock(
                start_date=pendulum.datetime(2020, 12, 1, 16, 20, 0, tz="Australia/Sydney"),
                interval=timedelta(days=1),
                parameter_defaults=param
            )
            for param in parameters
        ]
    )
    However when I deploy the flow to prefect server it runs 10 out of my 4000 odd params and schedules another 10 for tomorrow. Is there a limit to how many Interval clocks I can use with the prefect server schedule or potentially some other reason why it isn't queuing the full set of params to be run? Cheers, Phil
    p
    Jenny
    3 replies
    Copy to Clipboard