Perhaps my Google search skills aren't quite up to the task, but I'm trying to find the IP range that Prefect Cloud uses to communicate with other services. Right now, we were running into difficulties using the AirbyteConnectionTask without opening our AWS Security Group up to public access for the task to work which we aren't super excited about. I have the logs added in the thread, but in summary, it fails and shows the below stack trace when we only have port 22 open to a specific IP range and it succeeds when we open up the SG. Has anybody else run into this using the AirbyteConnectionTask?

Prefect Cloud itself wouldn’t communicate to other services. This would all occur on the Prefect Agent that picked up the flow. For instance, in my environment I use the Kubernetes agent which gets the egress IP I have configured for the cluster. What type of agent are you using?

We are using the local agent. This was my understanding of the communication mechanism in Prefect as well. With that, what should we do? Because it does work when we open the security group and it doesn't work when we only have it open to port 22. Is there any other information I can send over or details on this that would provide additional context?

If I am understand correctly, that local agent had to be installed somewhere (your laptop for dev, a server or vm, ec2, etc.). Where the local agent was installed would dictate the networking configuration. Do you happen to know where the local agent was installed/is running?

The local agent is installed on the EC2 instance hosting Airbyte

Gotcha, that’s interesting, It should be using that EC2's public ip address. I haven’t worked with AWS in a minute but I think you may want to look at how that EC2 instance is configured to see if it is part of a virtual network of some sort. That virtual network should be added to networking rules to allow it access. In this case since it sounds like the same server I’m wondering if it’s bouncing out and getting a public IP address

Just to make sure I'm understanding this, the security group attached to the EC2 instance which hosts Airbyte should have an ingress rule which allows traffic from the public IP address attached to the EC2 instance? Is that correct?

The discussion so far is right that the Agent machine just needs HTTP out in order to ping Prefect Cloud, and then it finds a flow to run and then execute it. So you shouldn’t need the IP ranges of Prefect Cloud in order to execute anything. I am confused why the Airbyte task would go through the internet if they are in the same box? Wouldn’t it hit a local port on the same machine? You can do

localhost

for the server host right?

I was gonna say my guess is because the URL is using the full public EC2 instance IP address.

Ahhh I understand what you are saying

But, that was a bit presumptuous of me lol. I could see labeled agents always on the airbyte hosts where localhost would be fine. So that may be the best first step. Sorry for any confusion