Hello - has anyone successfully used

prefect-airbyte

lately? I think Airbyte deprecated an old API address endpoint for health checks that the package uses, and every execution fails because of it

Hey Eric catjam What are your logs saying? I have had it running hourly with no issues today. We are using self hosted airbyte though, so maybe I missed an airbyte update 😵 .

I'm using self hosted myself too. Here's my setup: I have a simple setup:

ashby_sync_connection = AirbyteConnection.load("airbyte-connection-ashby-core")
sync_result = run_connection_sync(airbyte_connection=ashby_sync_connection)

I get this error when trying to trigger

File "/Users/albanese/prefect/venv/lib/python3.10/site-packages/prefect_airbyte/client.py", line 58, in check_health_status
    response.raise_for_status()
  File "/Users/albanese/prefect/venv/lib/python3.10/site-packages/httpx/_models.py", line 761, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url '<https://airbyte.myserver.com:8006/api/v1/health/>'

I think what is wrong is that the Airbyte connection is creating a base URL of

<https://airbyte.myserver.com:8006/api/v1/>

. However the Airbyte docs show the API address for a health check as

<https://api.airbyte.com/health>

and lacks any sort of

/api/v1

prefix

what version of Prefect-Airbyte you using

dont think I specified in my requirements.txt.... should I?

Sorry I can't type haha, I think

airbyte-connection-ashby-core

is wrong. Did you create that block in Prefect Cloud, and are loading it in basically?

Yeah created in Prefect Cloud

are you hosting Airbyte locally, or in a VM somewhere?

ECS. In the Prefect Server Block I tried leaving the

API Version

part blank but then it returns an error of

'404 Not Found' for url '<https://airbyte.xxx.com:8006/api//health/>'

with the api placeholder still

our server block is:

server = AirbyteServer(
        username=settings.AIRBYTE_USERNAME,
        password=settings.AIRBYTE_PASSWORD,
        server_host=server_host,
        server_port=8000,
        api_version="v1",
    )

So v1 is right. but let's backup. You are receiving

File "/Users/albanese/prefect/venv/lib/python3.10/site-packages/prefect_airbyte/client.py", line 58, in check_health_status
    response.raise_for_status()
  File "/Users/albanese/prefect/venv/lib/python3.10/site-packages/httpx/_models.py", line 761, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url '<https://airbyte.myserver.com:8006/api/v1/health/>'

I feel like that makes sense though, unless you redacted it, which is totally fine/makes sense. Shouldn't

myserver

be your VMs IP?

Yeah, we have a load balancer in front of it. Might be something with that perhaps? Let me try a direct IP request

Just thinking out-loud... We are in GCP, and have a VPC set up for ours. I remember I think I received this or something similar, and it was definitely related to me setting up VPC. I don't think this is firewall related, because it would return a connection dropped error I think.

yeah same issue. Trying

<http://10.xxxx/health>

works but

<http://10.xxxx:8006/api/v1/health>

returns a 404 🤔

So that test is you just navigating to it in browser, right?

GET request in Postman

got it

I totally appreciate it! It's definitely a curious one

I am convinced that there is an issue with block, I have idea to test that too. lemme send over snippet one sec for u to test

Yep

fill in values and try this, lemme know if u have any questions on any of the inputs. It's basically doing an 'anonyomous block' or idk what terminology is. Don't commit to VC or anything, since it will have passwords/etc.. just for a test

Hm so that seemed OK

(venv) ➜  prefect_airbyte git:(main) ✗ python flow_airbyte.py
14:50:09.100 | INFO    | prefect.engine - Created flow run 'precise-myna' for flow 'airbyte-syncs'
14:50:09.102 | INFO    | Flow run 'precise-myna' - View at <https://app.prefect.cloud/account/x>
14:50:10.670 | INFO    | Flow run 'precise-myna' - Created subflow run 'mellow-rabbit' for flow 'run-connection-sync'
14:50:10.671 | INFO    | Flow run 'mellow-rabbit' - View at <https://app.prefect.cloud/account/x>
14:50:11.213 | INFO    | Flow run 'mellow-rabbit' - Created task run 'trigger-0' for task 'trigger'
14:50:11.214 | INFO    | Flow run 'mellow-rabbit' - Executing 'trigger-0' immediately...
14:50:11.584 | INFO    | Task run 'trigger-0' - Triggering Airbyte Connection f8e3d2e8-728a-46f2-8ef8-26300af0c1f1, in workspace at '<http://localhost:8000/api/v1>'
14:50:12.506 | INFO    | Task run 'trigger-0' - Finished in state Completed()
14:50:12.696 | INFO    | Flow run 'mellow-rabbit' - Created task run 'wait_for_completion-0' for task 'wait_for_completion'
14:50:12.697 | INFO    | Flow run 'mellow-rabbit' - Executing 'wait_for_completion-0' immediately...
14:50:13.138 | INFO    | Task run 'wait_for_completion-0' - running

Yeah maybe... just curious why the Health check isn't failing here when

<http://localhost:8006/api/v1/health>

returns a 404

this ->

<http://localhost:8006/api/v1/health>

returns a 404 happens when you run in post man right? And just to clarify, you did replace localhost with your VM's/airbyte host IP right?

Yeah. Haven't been able to get anything but a 404 against that URL, regardless of IP

and just a get request? i'll try to hit ours, one sec

Ahhhh shoot I think I got it figured out

it worked for me

the Airbyte server block needs the port of the web server 8000, not the api endpoint 8006

ohhh I thought you were hardcoding and using that for some reason, it defualts in airbyte_prefect to 8000. Sorry i should have noticed that

all good my friend - appreciate the troubleshooting here

np! just a dog helping another dog out! reach out if anything else comes up, always happy to help !