Main BGP post

This is a continuation of the iBGP post. What isn't covered here, may be covered in iBGP, if it is something that affects both peering types.

eBGP

External BGP is peering between different AS numbers. New rules apply here and one of them is a default TTL of 1. This fact means, by default, you can only peer with a directly connected BGP router and on the physical interface. There are several extra configurations that can be used to get around this.

The topology has changed a bit from the iBGP. This is a multihomed setup, meaning 2 ISPs.

About the configuration. Full mesh iBGP with OSPF having full knowledge of all the internal routes. Loopback interfaces on iBGP along with dynamic peering on R1. In the eBGP peering, I want R1 and R3 to peer with both the ISP routers.

hostname R1 interface e1/0 ip address 1.1.1.1 255.255.255.252 no shut description R2 ! interface e1/1 ip address 1.1.1.5 255.255.255.252 no shut description R3 ! interface e2/0 ip address 100.1.1.2 255.255.255.252 no shut description R4 ! interface lo0 ip address 10.1.1.1 255.255.255.0 ! router ospf 1 router-id 1.1.1.1 passive-interface default no passive-interface e1/0 no passive-interface e1/1 no passive-interface lo0 network 1.1.1.0 0.0.0.7 area 0 network 10.1.1.1 0.0.0.0 area 0 ! router bgp 1 bgp router-id 1.1.1.1 neighbor local peer-group neighbor local route-reflector-client neighbor local remote-as 1 neighbor local update-source loopback0 bgp listen range 10.1.0.0/19 peer-group local bgp listen limit 10 neighbor external peer-group neighbor external ttl-security hops 2 neighbor external update-source Loopback0 neighbor 100.1.4.1 remote-as 2 neighbor 100.1.5.1 remote-as 3 neighbor 100.1.4.1 peer-group external neighbor 100.1.5.1 peer-group external ! ip route 100.1.4.1 255.255.255.255 100.1.1.1 ip route 100.1.5.1 255.255.255.255 1.1.1.6 ________________________________________ hostname R2 interface e1/0 ip address 1.1.1.2 255.255.255.252 no shut description R1 ! interface e1/2 ip address 1.1.1.9 255.255.255.252 no shut description R3 ! interface lo0 ip address 10.1.2.1 255.255.255.0 ! router ospf 1 router-id 2.2.2.2 passive-interface default no passive-interface e1/0 no passive-interface e1/1 no passive-interface lo0 network 10.1.2.1 0.0.0.0 area 0 network 1.1.1.0 0.0.0.3 area 0 network 1.1.1.8 0.0.0.3 area 0 ! router bgp 1 bgp router-id 2.2.2.2 neighbor local peer-group neighbor local remote-as 1 neighbor local update-source loopback0 neighbor 10.1.1.1 peer-group local ________________________________________ hostname R3 interface e1/1 ip address 1.1.1.6 255.255.255.252 no shut description R1 ! interface e1/2 ip address 1.1.1.10 255.255.255.252 no shut description R2 ! interface e2/0 ip address 100.1.1.10 255.255.255.252 no shut description R5 ! interface lo0 ip address 10.1.3.1 255.255.255.0 ! router ospf 1 router-id 3.3.3.3 passive-interface default no passive-interface e1/1 no passive-interface e1/2 no passive-interface lo0 network 1.1.1.4 0.0.0.3 area 0 network 1.1.1.8 0.0.0.3 area 0 network 10.1.3.1 0.0.0.0 area 0 ! router bgp 1 bgp router-id 3.3.3.3 neighbor local peer-group neighbor local remote-as 1 neighbor local update-source loopback0 neighbor 10.1.1.1 peer-group local neighbor external peer-group neighbor external ttl-security hops 2 neighbor external update-source Loopback0 neighbor 100.1.4.1 remote-as 2 neighbor 100.1.5.1 remote-as 3 neighbor 100.1.4.1 peer-group external neighbor 100.1.5.1 peer-group external ! ip route 100.1.5.1 255.255.255.255 100.1.1.9 ip route 100.1.4.1 255.255.255.255 1.1.1.5 ________________________________________ hostname R4 interface e2/0 ip address 100.1.1.1 255.255.255.252 no shut description R1 ! interface e3/0 ip address 100.1.1.5 255.255.255.252 no shut description R5 ! interface lo0 ip address 100.1.4.1 255.255.255.0 ! router bgp 2 bgp router-id 4.4.4.4 neighbor external peer-group neighbor external remote-as 1 neighbor external update-source lo0 neighbor external ttl-security hops 2 neighbor 10.1.1.1 peer-group external neighbor 10.1.3.1 peer-group external neighbor 100.1.5.1 remote-as 3 neighbor 100.1.5.1 update-source lo0 neighbor 100.1.5.1 ttl-security hops 2 ! ip route 10.1.1.1 255.255.255.255 100.1.1.2 ip route 10.1.3.1 255.255.255.255 100.1.1.2 ip route 100.1.5.1 255.255.255.255 100.1.1.6 ________________________________________ hostname R5 interface e2/0 ip address 100.1.1.9 255.255.255.252 no shut description R2 ! interface e3/0 ip address 100.1.1.6 255.255.255.252 no shut description R4 ! interface lo0 ip address 100.1.5.1 255.255.255.0 ! router bgp 3 bgp router-id 5.5.5.5 neighbor external peer-group neighbor external remote-as 1 neighbor external update-source lo0 neighbor external ttl-security hops 2 neighbor 10.1.1.1 peer-group external neighbor 10.1.3.1 peer-group external neighbor 100.1.4.1 remote-as 2 neighbor 100.1.4.1 update-source lo0 neighbor 100.1.4.1 ttl-security hops 2 ! ip route 10.1.1.1 255.255.255.255 100.1.1.10 ip route 10.1.3.1 255.255.255.255 100.1.1.10 ip route 100.1.4.1 255.255.255.255 100.1.1.5

Some issues have to be solved with our eBGP peers.

First off, a TTL of 1 means if R3 tries to peer with R4, it will send a packet to R1, which will then be dropped, as the TTL is now 0. I can either use eBGP multihop or TTL security. The multihop command simply increases the TTL for the packet to the amount specified on the neighbor statement. How many hops are specified here don't matter, as long as it's enough to reach the neighbor.

TTL security fits the same purpose and a little more. BGP doesn't check how many hops away the peers are, which means anyone on any side of the internet could potentially try to peer with a BGP router. This requires spoofing the IP address in the neighbor statement, guessing the TCP sequence and then you can start sending malicious TCP messages. What TTL security does, is set the TTL to 255 in the packet. With the TTL security hop x command, you specify how many maximum hops there are between the peers and this has to be done on both sides. BGP will then check the TTL value of the received packet, which now has to be equal or greater than 255 minus the specified hop count. Why this is considered secure, is because the BGP peers are likely not a lot of hops away from each other, but an attacker is very likely many more hops away. It is also considered nearly impossible to spoof the TTL, according to Cisco, which refers to the RFC 5082. The multihop and TTL commands don't work together, only one can be used.

We aren't completely done with the TTL issue yet. Even the directly connected neighbors, R1-R4 and R3-R5, will need some help to peer through their loopbacks. A TTL of 1, despite being directly, is not going to establish any relationship. The two options are increasing the TTL value to 2 or using disable-connected-check. The latter command does exactly as it says, disables a check in BGP for whether the neighbor IP is on the directly connected interface. I can't exactly figure out what the disable check does underneath the hood of BGP, but it pretty much means the BGP router is aware the peering is on a loopback. Regardless, it's arbitrary to me, as a person configuring the routers, how these things work in the core of the OS. But, it is a frustrating subject, because I can't find one clear answer to why the routers behave this way. I've seen documentation state that Cisco decrements the TTL in ingress, while other documentation states that TTL doesn't decrease before the packet is rewritten, meaning on the egress interface. Now, is it considered egress interface when the packet internally, on the router goes from the IP on the physical interface to the loopback IP? If so, then here comes the kicker:

Now what is shown here, is that I apply a TTL of 2 hops, which is the minimum required to establish peering on loopbacks through directly connected routers. But I am also getting connection to R5 from R1, which is R1>R3>R5 and by the logic of the ingress TTL decrement, I should need a 3 hop configuration. I am going to leave the TTL here, because it is an arbitrary issue and fact is, I could use 2 or 3 hops, as long as the connection is there.

I have chosen to use TTL security as it accomplishes the same as multihop and the disable check, while also providing security. There is also the semi issue that, should my R1/R3 routers lose their directly connected link, but R4/R5 are still up, they'd have no way of establishing/keeping their peering session, as their TTL is 1 and they're no longer directly connected, which ironically is a requirement to using the disable directly connected command. It'd also only be one peer on each router I could use the disable check command for, so this way I also save a whole line of configuration in my peer-group!

Next issue..

Because the next-hop in iBGP is not modified, I have to use a method for R3 to know how to reach the IP address of R4, and R1 to R5. To fix this, I can either create the routing information needed for the the iBGP routers to reach the ISPs or I can use a build-in BGP feature called Next-Hop Self. Next-hop self is a command that tells iBGP peers to reach their eBGP neighbor through itself. Applied to R1, R3 now knows that to get to R4, it has to go through R1, because R1 is telling R3 that whatever you're peering with externally, just use my IP as next-hop. iBGP has the full mesh rule, so it has no default need to change the next-hop address when routers are peering the same AS.

I can also use a route-map to set the next-hop address, but that is for another topic where I get more specifically into next-hop processing. When the next-hop manipulation begins, it should become easier to see why BGP is not a routing protocol. BGP can receive route information and be told to use another route to the destination, than through who advertised it.

My choice here is to use static routes. I create 2 static routes on each router and this creates the information needed for all routes to use the correct, unmodified next-hop address. Now, there is a design issue with my routing configuration. If I shut either of the eBGP links, then that ISP peer will be lost, because it can't route through the other ISP.

This is the routing required when using loopback interfaces:

I chose to use static routes for simplicity. Peering with an IGP may also be an option with the ISP. Either way, there are design complications which need to be solved, but those are actually from the perspective of the ISP(s). In most cases the ISP will likely tell you what you're options are to peer with them, in order to establish BGP.

Before getting into the issues of this multihome peering setup and what differences can be done, there is the question of whether or not it is even desirable of making it more redundant. This is a question of money, complexity and needs, something which I won't/can't get into here. It's more of a use case scenario, like I have two datacenters and I host a lot of webpages, so perhaps I'd like to have a dual-multi homed setup where I load balance 50/50 to have the least amount of webpages affected if one site fails kind-of-scenario. Practical out of the way.

What is the issue with rerouting the bgp through the ISPs. To start with, the rerouting would only matter if it is the link that fails. If the router itself fails, the rerouting is moot. Cables do get cut over from time to time or maybe an SFP fails.

This is the actual issue, though:

On R4 and R5 I had added their respective networks used in their BGP peering, so they now advertise it and R1/R3 receive them, since we're peering BGP. If the link between R1 and R4 fails, R1 could try and send the traffic through R3, since it has learned a route through BGP, but on R3 there is a static route pointing back to R1. The static route is preferred due to the distance metric. The solutions or alternatives are plenty. Floating static, IGP peering with ISP, other kind of routing setups, physical interface peering instead of loopback and probably some ways I don't know about. However, it is for another post because I'd rather try and explore this with a use case that I can more easily relate to, than just go through every technical way it may be possible. Perhaps I'll take some inspiration from my own place of work and what solution our ISP(s) have offered.

A small note about the AD of the BGP routes. I had manually applied a distance of 120 for external routes to this BGP and I simply forgotten to remove it. By default external routes will be advertised with a distance of 20, which is less than any IGP (except static), something that is worth keeping in mind as well.

Miscellaneous

I mentioned early something about dynamic peering and it is in the configuration tab as well. There isn't really a use for dynamic peering, but I wanted it down in a post so there'd be a referral place. Dynamic peering is an option for BGP to avoid manually configuring every neighbor. Couple that with route reflection and the amount of configuration can drastically decrease. Think DMVPN, peering with possibly hundreds of routers, but instead of typing every neighbor, you just input the subnet range in the tunnel. It's just one command needed for it to actually work bgp listen range 10.1.0.0/19 peer-group local. What it does is simply that the router now listens to any BGP requests coming from peers with an IP in the configured scope. It can only be applied on one side, as it doesn't proactively send messages in the scope. Another thing is, it has to be applied with a peer-group, which is actually pretty smart anyway. In the peer-group I have applied it to, I have only specified the remote-as 1 command, which means any eBGP peers wont be able to try and establish a connection dynamically.

Filtering transit traffic is something that is likely very desirable to do when peering BGP with the ISP. Because you are now peering with the ISP, you are also part of the global BGP routing table. This means internet traffic can potentially travel through your routers. There exists different solutions to completely avoid this, though. Using an ACL with regular expression, tags, prefix list, but it is something I'll save for a post about BGP community.

In regards to BGP peering security there is also the option of applying authentication. This may be seen as an alternative to the TTL security. The issue is it can also be exploited to overload the CPU of the router. Everytime the router receives an authentication request, it has to hash the password to see if it matches the MD5. Send a lot of passwords and you have a DoS attack.

This should complete basic eBGP stuff. There are still plenty of more advanced options, but a lot of them lean into subjects which kind of need to be explained in order to understand the the options.