I've asked this question in the Airbyte Slack, but got no traction so far, has anybody been using Airbyte + Prefect and know what the best practice is for communicating with the API for a deployed version of Airbyte? Hey folks I have an instance of Airbyte deployed on GCP following the tutorial below. https://docs.airbyte.com/deploying-airbyte/on-gcp-compute-engine#connect-to-airbyte This uses ssh to connect to the instance and use it in my browser via

localhost:8000

. I'm trying to connect to the instance (to make an API call) through a Python script, but I have no idea how to think about the authentication layer. How do I send an API call to the server? The documentation suggests not to open the instance up to HTTP traffic but I'm guessing that's a requirement to be able to communicate with the API?

Pinging @Nate who does our Airbyte work for enterprise clients. My current conclusion is the same as yours, but we’ll see what Nate has to say

Thanks @Kevin Kho Update from my side, I've actually got it working locally by sshing into the instance and making it available on localhost (good news). The next test is to see whether it works in

dev

when I run it with Prefect Cloud. The agent is set up on a worker in the same network on GCP so I'm hoping that the fact they share a private network means they MAY be accessible without exposing ourselves to any outside traffic. I'll report back.

well, looks like you got it under control lol

since the calls will originate from where the flows are running, you just need to make sure your runtime is in the same network space as your airbyte instance, e.g. I have some flows running on AWS EKS where the nodes are in same private subnet as the EC2 running airbyte I'm assuming you're not on airbyte cloud here, I'm less familiar with using that alongside prefect. Does that answer your question?

Thanks @Nate that confirms what I thought, but I just need to get around testing it 😄 We're hosting Airbyte ourselves in the same subnet as the Prefect so yeah, not on Airbyte Cloud and we're hoping that will work like you're saying.