David Anderson

    David Anderson

    3 months ago
    is anyone who uses
    AirbyteConnectionTask
    seeing a new error starting ~last week? all of my runs are failing with
    Error during execution of task: KeyError('schedule')
    . haven't changed a thing in my prefect flow configuration. thought maybe it was related to an airbyte upgrade (im running the self-hosted version), but im struggling to troubleshoot. any ideas?
    Kevin Kho

    Kevin Kho

    3 months ago
    Hey David, can I see the traceback you get?
    David Anderson

    David Anderson

    3 months ago
    sure thing @Kevin Kho
    Traceback (most recent call last):
      File "/home/ec2-user/.local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 884, in get_task_run_state
        logger=self.logger,
      File "/home/ec2-user/.local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
        return task.run(*args, **kwargs)  # type: ignore
      File "/home/ec2-user/.local/lib/python3.7/site-packages/prefect/utilities/tasks.py", line 456, in method
        return run_method(self, *args, **kwargs)
      File "/home/ec2-user/.local/lib/python3.7/site-packages/prefect/tasks/airbyte/airbyte.py", line 341, in run
        session, airbyte_base_url, connection_id
      File "/home/ec2-user/.local/lib/python3.7/site-packages/prefect/tasks/airbyte/airbyte.py", line 181, in _get_connection_status
        schedule = response.json()["schedule"]
    KeyError: 'schedule'
    Kevin Kho

    Kevin Kho

    3 months ago
    What is your Prefect version?
    David Anderson

    David Anderson

    3 months ago
    1.2.2
    Kevin Kho

    Kevin Kho

    3 months ago
    Ok so the code looks right. Your error is from this line which hits this endpoint, which does contain a
    schedule
    in the response. The only scenario I can see this happening in is if you get a non-200 status code back. We could add a guard against that, but for you, is this the first time you used it or has it worked before? Because I think there’s a chance you gave an object id that may not exist?
    David Anderson

    David Anderson

    3 months ago
    @Kevin Kho these three scheduled flows were running fine for a few weeks before all failing about a week back.
    Kevin Kho

    Kevin Kho

    3 months ago
    Gotcha I guess they have updated their API. Will look into it
    David Anderson

    David Anderson

    3 months ago
    hey @Kevin Kho, wanted to share a new piece of info on this. im now getting a new error: Error during execution of task:
    AttributeError("'AirbyteConnectionTask' object has no attribute 'stream_output'")
    . the relevant section of the log is below. im guessing the issue is as you suspected -- something changing with the airbyte api. the new error started presenting after i updated airbyte to 0.39.31-alpha (the newest release as-of yesterday).
    Task 'AirbyteConnectionTask': Exception encountered during task execution!
    Traceback (most recent call last):
      File "/home/ec2-user/.local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 884, in get_task_run_state
        logger=self.logger,
      File "/home/ec2-user/.local/lib/python3.7/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
        return task.run(*args, **kwargs)  # type: ignore
      File "/home/ec2-user/.local/lib/python3.7/site-packages/prefect/utilities/tasks.py", line 456, in method
        return run_method(self, *args, **kwargs)
      File "/home/ec2-user/.local/lib/python3.7/site-packages/prefect/tasks/airbyte/airbyte.py", line 341, in run
        session, airbyte_base_url, connection_id
      File "/home/ec2-user/.local/lib/python3.7/site-packages/prefect/tasks/airbyte/airbyte.py", line 176, in _get_connection_status
        self.logger.log(level=self.stream_output, msg=response.json())
    AttributeError: 'AirbyteConnectionTask' object has no attribute 'stream_output'
    Kevin Kho

    Kevin Kho

    3 months ago
    Hey David, thanks for following up. The people with Airbbyte knowledge have just gotten back from vacation so I’ll forward this
    David Anderson

    David Anderson

    3 months ago
    🙏 much appreciated!
    Kevin Kho

    Kevin Kho

    3 months ago
    So looks like there was a fix for the schedule one here: https://github.com/PrefectHQ/prefect/commit/7ce5d1ad25f57bfbbc31f69101693a9c3a74073f
    will be out next release
    David Anderson

    David Anderson

    3 months ago
    awesome, thanks for the heads-up