Thread
#prefect-ui
    Blake Enyart

    Blake Enyart

    5 months ago
    Perhaps my Google search skills aren't quite up to the task, but I'm trying to find the IP range that Prefect Cloud uses to communicate with other services. Right now, we were running into difficulties using the AirbyteConnectionTask without opening our AWS Security Group up to public access for the task to work which we aren't super excited about. I have the logs added in the thread, but in summary, it fails and shows the below stack trace when we only have port 22 open to a specific IP range and it succeeds when we open up the SG. Has anybody else run into this using the AirbyteConnectionTask?
    Removed the EC2 IP for potential security impact at this time from the stack trace.
    Kevin Mullins

    Kevin Mullins

    5 months ago
    Prefect Cloud itself wouldn’t communicate to other services. This would all occur on the Prefect Agent that picked up the flow. For instance, in my environment I use the Kubernetes agent which gets the egress IP I have configured for the cluster. What type of agent are you using?
    Blake Enyart

    Blake Enyart

    5 months ago
    We are using the local agent. This was my understanding of the communication mechanism in Prefect as well. With that, what should we do? Because it does work when we open the security group and it doesn't work when we only have it open to port 22. Is there any other information I can send over or details on this that would provide additional context?
    It is specifically the health checks for Airbyte sent to the EC2 instance we have Airbyte hosted on that are failing. So when we have public communication it works fine, but when we shut it down to just port 22 it fails
    Kevin Mullins

    Kevin Mullins

    5 months ago
    If I am understand correctly, that local agent had to be installed somewhere (your laptop for dev, a server or vm, ec2, etc.). Where the local agent was installed would dictate the networking configuration. Do you happen to know where the local agent was installed/is running?
    Blake Enyart

    Blake Enyart

    5 months ago
    The local agent is installed on the EC2 instance hosting Airbyte
    Kevin Mullins

    Kevin Mullins

    5 months ago
    Gotcha, that’s interesting, It should be using that EC2's public ip address. I haven’t worked with AWS in a minute but I think you may want to look at how that EC2 instance is configured to see if it is part of a virtual network of some sort. That virtual network should be added to networking rules to allow it access. In this case since it sounds like the same server I’m wondering if it’s bouncing out and getting a public IP address
    As a test, you could find out what the public IP address is of the EC2 instance and add a rule to the security group allow it for that ip address to the necessary ports that Airbyte uses.
    Blake Enyart

    Blake Enyart

    5 months ago
    Just to make sure I'm understanding this, the security group attached to the EC2 instance which hosts Airbyte should have an ingress rule which allows traffic from the public IP address attached to the EC2 instance? Is that correct?
    Kevin Kho

    Kevin Kho

    5 months ago
    The discussion so far is right that the Agent machine just needs HTTP out in order to ping Prefect Cloud, and then it finds a flow to run and then execute it. So you shouldn’t need the IP ranges of Prefect Cloud in order to execute anything. I am confused why the Airbyte task would go through the internet if they are in the same box? Wouldn’t it hit a local port on the same machine? You can do
    localhost
    for the server host right?
    Kevin Mullins

    Kevin Mullins

    5 months ago
    I was gonna say my guess is because the URL is using the full public EC2 instance IP address.
    Yea, you could technically use “localhost”, but seems like there may be some scalability type decisions there, can you always assume airbyte will be on the same machine as the agent, do you need multiple agents/airbyte servers, etc
    Prolly should have led with that. I was thinking from the perspective of a solution that would work if it started on the same machine but moved to another or expanded in other ways, using vnet type rules would prolly be ideal between ec2 instances imho
    Kevin Kho

    Kevin Kho

    5 months ago
    Ahhh I understand what you are saying
    Kevin Mullins

    Kevin Mullins

    5 months ago
    But, that was a bit presumptuous of me lol. I could see labeled agents always on the airbyte hosts where localhost would be fine. So that may be the best first step. Sorry for any confusion