BGP PIC ( Prefix Independence Convergence)

PIC (Prefix Independent Convergence) is a feature to decrease the data plane convergence time. There are two flavors:

  • BGP PIC Core: decreases convergence time when a core router fails and your IGP has to find a new best path to your PE router.
  • BGP PIC Edge: decreases convergence time when a PE router fails and BGP has to switch to a different PE router.

PIC is best explained by showing it in action. We’ll use the following topology:

Bgp Pic Topology Ce Pe P Routers

Above, we have a small provider network in AS 2 with routers running iBGP. P2 is our route reflector in this network. We have two customers, CE1 and CE2. CE2 is advertising two prefixes in BGP. Within AS 2, we use OSPF as the IGP. I increased the cost on the interfaces of the P2 router so that P1 is the preferred path.

  • Configurations
  • CE1
  • CE2
  • P1
  • P2
  • PE1
  • PE2
  • PE3
  • PE4

Want to take a look for yourself? Here you will find the startup configuration of each device.

BGP PIC Core

PIC core helps to speed up convergence when there is a failure in the core of your network and the BGP next hop does not change.

Let’s have a look at PE1:

PE1#show ip bgp
BGP table version is 23, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>i  8.8.8.8/32       6.6.6.6                  0    100      0 8 i
 *>i  88.88.88.88/32   6.6.6.6                  0    100      0 8 i

Let’s focus on 8.8.8.8/32 and 88.88.88.88/32. These two prefixes were advertised by CE2 and PE1 learned them from P2 (our route reflector). The next hop for both prefixes is 6.6.6.6. When PE1 wants to reach CE2, our traffic path looks like this:

Bgp Pic Core Traffic Path Pe1 Ce2

How does 6.6.6.6 get resolved? Let’s have a look:

PE1#show ip route 6.6.6.6
Routing entry for 6.6.6.6/32
  Known via "ospf 1", distance 110, metric 3, type intra area
  Last update from 192.168.24.4 on GigabitEthernet0/1, 00:44:32 ago
  Routing Descriptor Blocks:
  * 192.168.24.4, from 6.6.6.6, 00:44:32 ago, via GigabitEthernet0/1
      Route metric is 3, traffic share count is 1

We learn about 6.6.6.6 through OSPF and to get to this next hop, we use next hop 192.168.24.4 through interface GigabitEthernet0/1. This is what gets installed in the FIB (CEF table):

PE1#show ip cef 6.6.6.6
6.6.6.6/32
  nexthop 192.168.24.4 GigabitEthernet0/1

What about those two prefixes? This is how they show up in the FIB:

PE1#show ip cef | include 8.8
8.8.8.8/32           192.168.24.4         GigabitEthernet0/1
88.88.88.88/32       192.168.24.4         GigabitEthernet0/1

Here’s a quick summary of the recursive routing we saw:

  • BGP: to get to 8.8.8.8/32 and 88.88.88.88/32 we use next hop 6.6.6.6.
  • IGP: to get to 6.6.6.6/32 we use next hop 192.168.24.4 with interface GigabitEthernet0/1.
  • FIB:
    • 8.8.8.8/32 via 192.168.24.4 interface GigabitEthernet0/1
    • 88.88.88.88/32 via 192.168.24.4 interface GigabitEthernet0/1

For each prefix you have learned, you find an entry like this in the FIB. We only have two prefixes but if you learned a million prefixes through BGP, you’ll see them here.

Disaster strikes our provider and suddenly P1 fails:

Bgp Pic Core P1 Failure

How does this impact PE1 and how it gets to CE2?

Let’s find out! To see what happens in real-time, I’ll create an access-list that matches the next hop (6.6.6.6) and our two prefixes (8.8.8.8/32 and 88.88.88.88/32). We attach the access-list to a debug:

PE1(config)#access-list 1 permit host 6.6.6.6
PE1(config)#access-list 1 permit host 8.8.8.8
PE1(config)#access-list 1 permit host 88.88.88.88

PE1#debug ip routing 1
IP routing debugging is on for access list 1

Let’s shut all interfaces of P1 now:

P1(config)#interface range GigabitEthernet 0/1 - 4
P1(config-if-range)#shutdown

Now it’s up to OSPF to figure out that P1 is gone and find a new path to 6.6.6.6:

PE1#
RT: updating ospf 6.6.6.6/32 (0x0)  :
    via 192.168.25.5 Gi0/2  0 1048578

RT: closer admin distance for 6.6.6.6, flushing 1 routes
RT: add 6.6.6.6/32 via 192.168.25.5, ospf metric [110/4]
RT: updating bgp 8.8.8.8/32 (0x0)  :
    via 6.6.6.6   0 1048577

RT: closer admin distance for 8.8.8.8, flushing 1 routes
RT: add 8.8.8.8/32 via 6.6.6.6, bgp metric [200/0]
RT: updating bgp 88.88.88.88/32 (0x0)  :
    via 6.6.6.6   0 1048577

RT: closer admin distance for 88.88.88.88, flushing 1 routes
RT: add 88.88.88.88/32 via 6.6.6.6, bgp metric [200/0]

We see that OSPF reconverges and installs 192.168.25.5 (P2) as the new next hop to get to 6.6.6.6/32. BGP is also updated. Let’s take a look at the BGP table:

PE1#show ip bgp
BGP table version is 33, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>i  8.8.8.8/32       6.6.6.6                  0    100      0 8 i
 *>i  88.88.88.88/32   6.6.6.6                  0    100      0 8 i

Nothing changed in the BGP table, this makes sense since our next hop did not change. Only the path to get to the next hop. Let’s take a look at OSPF:

PE1#show ip route 6.6.6.6
Routing entry for 6.6.6.6/32
  Known via "ospf 1", distance 110, metric 4, type intra area
  Last update from 192.168.25.5 on GigabitEthernet0/2, 00:02:06 ago
  Routing Descriptor Blocks:
  * 192.168.25.5, from 6.6.6.6, 00:02:06 ago, via GigabitEthernet0/2
      Route metric is 4, traffic share count is 1

We see that OSPF now uses 192.168.25.5 (P2) to get to 6.6.6.6/32 via the GigabitEthernet0/2 interface. Let’s take a look at the FIB for our next hop:

PE1#show ip cef 6.6.6.6
6.6.6.6/32
  nexthop 192.168.25.5 GigabitEthernet0/2

Above, we see the new next hop for 6.6.6.6/32 and interface in the FIB. Let’s take a look at the FIB entries for our two prefixes:

PE1#show ip cef | include 8.8    
8.8.8.8/32           192.168.25.5         GigabitEthernet0/2
88.88.88.88/32       192.168.25.5         GigabitEthernet0/2

The FIB entries for our two prefixes has changed as well to reflect the new next hop and outgoing interface. Think about this for a second…

Our router had to figure out a new path to get to 6.6.6.6/32 and update all entries in the FIB that used the old next hop (192.168.24.4) to the new next hop (192.168.25.5). With two prefixes, this is a piece of cake but what if we have one million prefixes in our FIB? They all have to get updated one-by-one and this takes a LONG time (minutes).

The problem here is how our FIB works. We use what we call a “flat” or “flattened” FIB structure:

Flat Fib

With a flat FIB, there is a one-to-one relationship between the prefix and the next hop / outgoing interface. When the next hop changes, the router walks through each prefix in the FIB to update the next hop. This is a time-consuming process that requires a lot of CPU cycles.

To improve this, we use a different structure for our FIB, called the hierarchical FIB:

Hierarchical Fib Pe1

Instead of a flat table, we use a hierarchical table where we also store the BGP next hops and the IGP next hops separately.

This has a huge advantage: when the IGP next hop changes, only the part of the FIB that is affected has to change:

Hierarchical Fib Failover

We change the pointer from the BGP path list from “PE3 via 192.168.24.4” to “PE3 via 192.168.25.5” and that’s it! We don’t have to touch the BGP prefixes or BGP path list since those did not change! That’s why we call it “prefix independent”.

With something like a million prefixes, this makes the difference between a convergence time of minutes to less than 200 ms. The convergence time depends on how fast your IGP converges to find the new next hop. Some other advantages are:

  • Reduced FIB memory requirements since a lot of information is shared.
  • Fewer CPU cycles needed because we don’t have to flatten all BGP prefixes when the IGP next hop changes.

The hierarchical FIB is supported on pretty much any Cisco platform. On Cisco IOS XR and NX-OS, it is the default. On Cisco IOS, we can enable it with a simple command:

PE1, PE2, PE3 & PE4
(config)#cef table output-chain build favor convergence-speed

It’s only a single command to enable the hierarchical FIB and PIC core but if you want to use this, it doesn’t end here. Since this relies on your IGP convergence time, there are some things you need to implement to speed up your convergence times like:

  • tuning your IGP timers
  • implementing something like BFD or FRR

Before we continue with PIC edge, let’s restore P1:

P1(config-if-range)#no shutdown

BGP PIC Edge

BGP PIC core helps when there is an issue in the core. What if one of your PE routers fails or a link between a PE and CE router?

When that happens, BGP has to reconverge and switch from one PE router to another PE router.



I’ll show you a “before” and “after” scenario of PIC edge from PE1’s perspective when PE3 fails. We use the exact same topology that we started with.

Without BGP PIC Edge

Let’s take a look at the BGP table of PE1:

PE1#show ip bgp

     Network          Next Hop            Metric LocPrf Weight Path
 *>i  8.8.8.8/32       6.6.6.6                  0    100      0 8 i
 *>i  88.88.88.88/32   6.6.6.6                  0    100      0 8 i

We see the 8.8.8.8/32 and 88.88.88.88/32 prefix that both use next hop 6.6.6.6 (PE3). Let’s check the OSPF routing table:

PE1#show ip route 6.6.6.6
Routing entry for 6.6.6.6/32
  Known via "ospf 1", distance 110, metric 3, type intra area
  Last update from 192.168.24.4 on GigabitEthernet0/1, 00:02:03 ago
  Routing Descriptor Blocks:
  * 192.168.24.4, from 6.6.6.6, 00:15:49 ago, via GigabitEthernet0/1
      Route metric is 3, traffic share count is 1

To get to 6.6.6.6/32, we use next hop 192.168.24.4. This gets installed in the FIB:

PE1#show ip cef 6.6.6.6
6.6.6.6/32
  nexthop 192.168.24.4 GigabitEthernet0/1

And this is how our prefixes look in the FIB:

PE1#show ip cef | include 8.8
8.8.8.8/32           192.168.24.4         GigabitEthernet0/1
88.88.88.88/32       192.168.24.4         GigabitEthernet0/1

PE1 uses the path through P1 to get to P3:

Bgp Pic Core Traffic Path Pe1 Ce2

Disaster strikes again…our provider recovered from the P1 failure but now their PE3 router fails:

Bgp Pic Core Pe3 Failure

On PE1, I enable this debug again:

PE1#debug ip routing 1
IP routing debugging is on for access list 1

And to simulate the PE3 failure, we’ll shut down all its interfaces:

PE3(config)#interface range GigabitEthernet 0/1 - 3
PE3(config-if-range)#shutdown

Here’s what we see on PE1:

PE1#
RT: del 6.6.6.6 via 192.168.24.4, ospf metric [110/3]
RT: delete subnet route to 6.6.6.6/32
RT: del 8.8.8.8 via 6.6.6.6, bgp metric [200/0]
RT: delete subnet route to 8.8.8.8/32
RT: del 88.88.88.88 via 6.6.6.6, bgp metric [200/0]
RT: delete subnet route to 88.88.88.88/32
RT: updating bgp 8.8.8.8/32 (0x0)  :
    via 7.7.7.7   0 1048577

RT: add 8.8.8.8/32 via 7.7.7.7, bgp metric [200/0]
RT: updating bgp 88.88.88.88/32 (0x0)  :
    via 7.7.7.7   0 1048577

RT: add 88.88.88.88/32 via 7.7.7.7, bgp metric [200/0]
RT: updating bgp 8.8.8.8/32 (0x0)  :
    via 7.7.7.7   0 1048577

RT: closer admin distance for 8.8.8.8, flushing 1 routes
RT: add 8.8.8.8/32 via 7.7.7.7, bgp metric [200/0]
RT: updating bgp 88.88.88.88/32 (0x0)  :
    via 7.7.7.7   0 1048577

RT: closer admin distance for 88.88.88.88, flushing 1 routes
RT: add 88.88.88.88/32 via 7.7.7.7, bgp metric [200/0]

OSPF figures out that 6.6.6.6/32 is gone so it gets removed from the routing table. BGP reconverges and figures out that 7.7.7.7 (PE4) is the new next hop.

Let’s take a look at the updated BGP table:

PE1#show ip bgp

     Network          Next Hop            Metric LocPrf Weight Path
 *>i  8.8.8.8/32       7.7.7.7                  0    100      0 8 i
 *>i  88.88.88.88/32   7.7.7.7                  0    100      0 8 i

We see that BGP now uses 7.7.7.7 as the next hop. How does it get resolved?

PE1#show ip route 7.7.7.7
Routing entry for 7.7.7.7/32
  Known via "ospf 1", distance 110, metric 3, type intra area
  Last update from 192.168.24.4 on GigabitEthernet0/1, 00:07:51 ago
  Routing Descriptor Blocks:
  * 192.168.24.4, from 7.7.7.7, 00:21:37 ago, via GigabitEthernet0/1
      Route metric is 3, traffic share count is 1

To get to 7.7.7.7/32 we have to go to 192.168.24.4 (P1) with outgoing interface GigabitEthernet0/1. Here’s how this gets installed in the FIB:

PE1#show ip cef 7.7.7.7
7.7.7.7/32
  nexthop 192.168.24.4 GigabitEthernet0/1
PE1#show ip cef | include 8.8
8.8.8.8/32           192.168.24.4         GigabitEthernet0/1
88.88.88.88/32       192.168.24.4         GigabitEthernet0/1

This is all looking good but it’s a pretty slow process.  BGP has to reconverge since the path through PE3 no longer exists. It will take some time before P2 (the route reflector) figures out that PE3 is gone and advertises the new path through PE4 (7.7.7.7) to PE1.

Let’s see how we can improve this. Let’s recover PE3 before we continue:

PE3(config)#interface range GigabitEthernet 0/1 - 3
PE3(config-if-range)#no shutdown

With BGP PIC Edge

We can configure BGP to pre-install a backup next hop. To do that, we need a second next hop. That’s a problem right now. P2 is our route reflector and has two paths:

P2#show ip bgp

     Network          Next Hop            Metric LocPrf Weight Path
 *>i  8.8.8.8/32       6.6.6.6                  0    100      0 8 i
 *>i  88.88.88.88/32   6.6.6.6                  0    100      0 8 i

BGP only advertises the best path so that’s why PE1 only has an entry through PE3 to get to these prefixes. We can change this behavior with the BGP additional paths feature. Let’s enable this on P2 and PE1 so that PE1 receives two paths.

P2 has to send additional paths:

P2(config)#router bgp 2
P2(config-router)#address-family ipv4
P2(config-router-af)#bgp additional-paths select all
P2(config-router-af)#neighbor 2.2.2.2 additional-paths send
P2(config-router-af)#neighbor 2.2.2.2 advertise additional-paths all

And PE1 needs to receive them:

PE1(config)#router bgp 2
PE1(config-router)#address-family ipv4
PE1(config-router-af)#neighbor 5.5.5.5 additional-paths receive

Now let’s take a look at PE1:

PE1#show ip bgp 

     Network          Next Hop            Metric LocPrf Weight Path
 *>i  8.8.8.8/32       6.6.6.6                  0    100      0 8 i
 * i                   7.7.7.7                  0    100      0 8 i
 *>i  88.88.88.88/32   6.6.6.6                  0    100      0 8 i
 * i                   7.7.7.7                  0    100      0 8 i

Very nice, we see two paths…one through PE3 and the other one through PE4. Only the path through PE3 is installed though:

PE1#show ip bgp 8.8.8.8
BGP routing table entry for 8.8.8.8/32, version 64
Paths: (2 available, best #1, table default)
  Advertised to update-groups:
     2         
  Refresh Epoch 1
  8
    6.6.6.6 (metric 3) from 5.5.5.5 (5.5.5.5)
      Origin IGP, metric 0, localpref 100, valid, internal, best
      Originator: 6.6.6.6, Cluster list: 5.5.5.5
      rx pathid: 0x0, tx pathid: 0x0
  Refresh Epoch 1
  8
    7.7.7.7 (metric 3) from 5.5.5.5 (5.5.5.5)
      Origin IGP, metric 0, localpref 100, valid, internal
      Originator: 7.7.7.7, Cluster list: 5.5.5.5
      rx pathid: 0x1, tx pathid: 0

We can change this with the bgp additional-paths command. Let’s configure PE1 to install the second path as a backup:

PE1(config)#router bgp 2
PE1(config-router)#address-family ipv4
PE1(config-router-af)#bgp additional-paths install

Let’s check the BGP table again:

PE1#show ip bgp
BGP table version is 67, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>i  8.8.8.8/32       6.6.6.6                  0    100      0 8 i
 *bi                   7.7.7.7                  0    100      0 8 i
 *>i  88.88.88.88/32   6.6.6.6                  0    100      0 8 i
 *bi                   7.7.7.7                  0    100      0 8 i

Note the b that stands for backup path. You can also see this if you take a closer look at a single prefix:

PE1#show ip bgp 8.8.8.8
BGP routing table entry for 8.8.8.8/32, version 66
Paths: (2 available, best #1, table default)
  Additional-path-install
  Advertised to update-groups:
     2         
  Refresh Epoch 1
  8
    6.6.6.6 (metric 3) from 5.5.5.5 (5.5.5.5)
      Origin IGP, metric 0, localpref 100, valid, internal, best
      Originator: 6.6.6.6, Cluster list: 5.5.5.5
      rx pathid: 0x0, tx pathid: 0x0
  Refresh Epoch 1
  8
    7.7.7.7 (metric 3) from 5.5.5.5 (5.5.5.5)
      Origin IGP, metric 0, localpref 100, valid, internal, backup/repair
      Originator: 7.7.7.7, Cluster list: 5.5.5.5
      rx pathid: 0x1, tx pathid: 0

This entry gets installed in the FIB right away. Take a look here:

PE1#show ip cef 8.8.8.8 detail 
8.8.8.8/32, epoch 0, flags [rib only nolabel, rib defined all labels]
  recursive via 6.6.6.6
    nexthop 192.168.24.4 GigabitEthernet0/1
  recursive via 7.7.7.7, repair
    nexthop 192.168.24.4 GigabitEthernet0/1

Once the primary next hop (6.6.6.6) fails, we can forward to the backup (7.7.7.7) right away!

Want to take a look for yourself? Here you will find the final configuration of each device.

Conclusion

You have now learned what BGP PIC (Prefix Independent Convergence) is.

  • BGP PIC (Prefix Independent Convergence) helps to decrease the data plane convergence time.
  • There are two flavors:
    • BGP PIC Core:
      • helps when a core router fails, the BGP next hop does not change and your IGP has to find a new path.
      • BGP prefixes are installed in a flat FIB with the recursed next hop and outgoing interface. When the next hop changes, the router has to go through all prefixes one-by-one in the FIB and change the next hop.
      • A hierarchical FIB stores the BGP and IGP path list. When an IGP next hop changes, you don’t have to re-flatten the entire table again. This decreases convergence time in the data plane.
    • BGP PIC Edge:
      • helps when a PE router fails and BGP has to find a new next hop.
      • BGP can pre-install a backup next hop for another BGP path in the FIB.
      • When using iBGP and a route reflector, you’ll need to use the BGP additional paths feature.

I hope you enjoyed this lesson. If you have any questions feel free to leave a comment!

Unit 1: Introduction to BGP

Unit 2: BGP Neighbor Adjacency

Unit 3: BGP Attributes

Unit 4: BGP Communities

Unit 5: BGP Filtering

Unit 6: Advanced BGP Features

Unit 7: BGP Convergence