This blog will delve into multiple Juniper features to enhance failover convergence, often referred to as BGP-PIC (Prefix-Independent Convergence).
This set of features improves convergence speed in BGP-based networks. It achieves this by precomputing backup paths for BGP prefixes, minimizing disruptions in the event of network failures such as link or router outages. This rapid switching to backup paths improves network availability and service reliability.
This blog will explain how PIC works for different services such as: Internet, L3VPN and different transport: Directly connected, IGP, BGP-LU, BGP-CT.
BGP Fast Reroute (FRR) can be intricate and may appear to deviate from BGP's original intent, which was deliberately crafted as a slower protocol for efficiently managing extensive route tables in large-scale IP environments. Its primary aim was to handle millions of routes without disrupting network equipment each time the IP topology undergoes changes.
BGP PIC supports multiprotocol BGP (MP-BGP) IPv4 or IPv6 Unicast and VPN network layer reachability information (NLRI) resolved using any of these protocols:
OSPF
IS-IS
LDP
RSVP
SR-TE
Flex-Algo
BGP-LU / BGP-CT
To clarify, BGP-PIC does not have SAFI, AFI, or NLRI associated with it. Instead, it's a technique used to enhance the performance of the forwarding convergence.
The key to optimizing the FIB for achieving Fast Reroute (FRR) capabilities lies in the hierarchical structure of next hops implemented within the hardware Packet Forwarding Engine (PFE), where the FIB resides. These structures are commonly referred to as indirect next hops or chained composite next hops. These techniques are used to realize BGP PIC (Prefix scale Independent Convergence) to illustrate this concept.
The next hop is programmed with local 'Weight' values, used to select a backup path when a link failure occurs, with a preference for lower weight values as the preferred route. In situations where routes have the same weight value, load balancing becomes possible.
In Junos, BGP PIC will categorize next hop as follows:
0x1 indicates the primary path with active next hops.
Equivalent of "1" in decimal - 0x4000 indicates the backup path with passive next hops.
Equivalent of "16384" in decimal
Junos features use the following four knobs, to enable BGP PIC for various use cases. Multiple options, each for different design types to address specific BGP FRR requirements:
Protect Core for service or transport RIB, Master table or VRF. This option installs a precomputed next hop in the RIB and FIB with higher weight as a backup.
set protocols bgp group <group_name> family inet unicast protection
set protocols bgp group <group_name> family inet6 unicast protection
set protocols bgp group <group_name> family inet labeled-unicast protection
set protocols bgp group <group_name> family inet6 labeled-unicast protection
set protocols bgp group <group_name> family inet transport protection
set protocols bgp group <group_name> family inet6 transport protection
For BGP-CT, only this knob is required. Both iBGP and eBGP is supported
Preserve-Nexthop-Hierarchy is required when the indirect-next-hop is behind another indirect next-hop.
It is important to note that in Juniper routers, Next-hop install, and protection next-hops utilize a separate memory block than the FIB, therefor no impact on the maximum FIB route scale. In Juniper systems such as MX and PTX routers, Next-hop supports unlimited number of next-hops in a chain.
Juniper silicon devices are uniquely positioned to support these capabilities. This is due to programmable next-hop processor and at each level in our case, you can enable protection.
Note: that other 3rd party systems are limited to support the feature for unlabeled transport or for BGP-LU only.
In Junos, different types of next hops are used in the context of routing and forwarding. Each type serves a specific purpose and has distinct characteristics. Here's an explanation of the common types of next hops in Juniper's routing platform:
Figure 1: Hierarchical Routing Structure
Note: Next-Hops occupying memory led to constraints and limitations. However, that's not the case with Juniper devices such as MX Trio and PTX Express silicon.
Juniper devices stand out in their ability to support these features at high scale, thanks to the programmable next-hop processor. Unlike other silicon devices, which support only three next-hops in a chain, Juniper devices can accommodate an unlimited number of next-hops in a chain. Additionally, you have the option to enable protection at each level.
Protocol Next Hop:
The Protocol Next Hop (PNH) is used to determine the forwarding next-hop. A next-hop as specified by a BGP update for the prefixes it is advertising. BGP Route Resolver recurses through protocol next-hop of the prefix to find its forwarding next-hop.
Composite Next Hop:
A Composite Next Hop (CNH) in Junos is a collection of Next Hop IDs controlled by a composition function that determines their actions. CNH simplifies label management by moving service labels to the top hierarchy. It's used for aggregating multiple next hops into one entity, useful for scenarios like Equal-Cost Multi-Path (ECMP) or Weighted ECMP routing. This optimization enhances network efficiency by reducing memory usage compared to unicast next-hops and creating a single next-hop for multiple paths. In L3VPN cases, when multiple routes are advertised with different labels, CNH helps create a single indirect next-hop shared by all routes.
Indirect Next Hop:
An indirect next-hop is a pseudo next-hop that references one of the actual forwarding next-hops, such as unicast, unilist, indexed, or others. It serves as a way to indirect or refer to one of these "real" forwarding next-hops, allowing for more flexible and efficient routing and forwarding options.
Unilist Next Hop:
List of Unicast and/or Unilist next-hops. When a packet is forwarded to a Unilist next-hop, it is sent to one of the entries within the Unilist. This particular next-hop type is commonly utilized for load balancing purposes, allowing the router to distribute traffic across multiple paths or destinations, thereby optimizing network performance and redundancy. Unilist supports multiple applications: ECMP, WECMP, FRR, AE.
Unicast Next Hop:
Direct physical next hop, containing the outgoing interface and full encapsulation such as full MPLS label information specified by the family type.
These different types of next hops enable Juniper routers and switches to handle various routing scenarios, including load balancing, ECMP, and direct forwarding to interfaces. The choice of next hop type depends on the network design and routing requirements for specific routes and destinations.
Introduced in Junos 13.2, with subsequent improvements for L3VPN in Junos 15.1, RSVP in Junos 16.1, and EVPN-VXLAN in Junos 22.4R1.
Figure 2: Fast reroute using backup path towards PE3
By default, even though the route to PE3 is learned via both P1 and P2, only the best route will be installed in the routing and forwarding table (marked with *). In this example, the path via ae11.101 is chosen.
mnayman@PE1> show route 192.168.101.3
inet.0: 36 destinations, 49 routes (36 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both
192.168.101.3/32 *[BGP/170] 00:07:08, localpref 100
AS path: 718 718 I, validation-state: unverified
> to 10.101.1.1 via ae11.101, Push 55
[BGP/170] 00:00:01, localpref 100
AS path: 718 718 I, validation-state: unverified
> to 10.102.1.1 via ae21.101, Push 54
RPD will program the PFE with one of the two routes in the forwarding table:
mnayman@PE1> show route forwarding-table destination 192.168.0.3
Routing table: default.inet
Internet:
Destination Type RtRef Next hop Type Index NhRef Netif
192.168.0.3/32 user 0 10.1.1.1 ucst 702 13 ae11.100
In the absence of BGP PIC, the following sequence of events would occur in the event of a failure:
When the interface goes down, it triggers the BGP session to go down.
RPD will be tasked with updating the routing table to reflect the changes for a best new path to be installed.
Subsequently, RPD will instruct the PFE to program the new route into its forwarding table.
These back-and-forth tasks of revoking and installing, followed by a BGP state change, can take more time than is required for some specific network environments.
BGP PIC improves and achieves BGP FRR with Primary and Backup Paths:
Preinstalled paths in the routing table reduce convergence times to milliseconds.
Two routes will be installed in the routing table, marked with an asterisk, indicating that the route is both the active and the last active route to a BGP prefix.
The primary path serves as the chosen route for routing traffic, while the backup path is a backup used in case of a link or router failure.
mnayman@PE1> show route 192.168.101.3
inet.0: 36 destinations, 49 routes (36 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both
192.168.101.3/32 *[BGP/170] 00:00:06, localpref 100
AS path: 718 718 I, validation-state: unverified
> to 10.101.1.1 via ae11.101, Push 55
to 10.102.1.1 via ae21.101, Push 54
[BGP/170] 00:00:47, localpref 100
AS path: 718 718 I, validation-state: unverified
> to 10.102.1.1 via ae21.101, Push 54
Let's dive deeper with an extensive set of commands to view the weights:
mnayman@PE1>show route forwarding-table destination 200.0.0.1/32 table VRF-A
Routing table: VRF-A.inet
Internet:
Destination Type RtRef Next hop Type Index NhRef Netif
200.0.0.1/32 user 0 comp 1048582 13
Let's enhance PE1's intelligence by enabling 'protect core' at the VRF level.
mnayman@PE1> show route forwarding-table destination 200.0.0.1/32 table VRF-A
Routing table: VRF-A.inet
Internet:
Destination Type RtRef Next hop Type Index NhRef Netif
200.0.0.1/32 user 0 ulst 1048582 12
comp 1048578 4
comp 1048585 3
With the extensive command, we can determine the preferred next-hop as well as the preferred unicast next-hop.
Introduced in Junos 13.2, this feature adds a pre-computed protection path into the PFE. In the event that a PE-CE link becomes unusable for forwarding, the PFE will reroute through another path without having to wait for the router or the protocols to provide updated forwarding information.
In addition to the protect core feature, when we examine the best route towards CE1, only one active route is installed in the routing table based on BGP path selection- the shortest path.
mnayman@PE1> show route 100.0.0.1/32
VRF-A.inet.0: 34 destinations, 59 routes (31 active, 0 holddown, 3 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both
100.0.0.1/32 *[BGP/170] 00:07:02, localpref 100
AS path: 101 I, validation-state: unverified
> to 172.18.1.1 via ge-0/0/0.0
[BGP/170] 00:08:13, localpref 100, from 192.168.0.210
AS path: 101 I, validation-state: unverified
> to 10.101.1.1 via ae11.101, Push 16, Push 54(top)
to 10.102.1.1 via ae21.101, Push 16, Push 53(top)
It becomes even clearer with an extensive command. The source 172.18.1.1 is the eBGP peer CE1, advertising the active route marked with an asterisk, while the inactive route is learned from the route-reflector source 192.168.0.210 with the reason for inactivity:
mnayman@PE1> show route 100.0.0.1/32 extensive
VRF-A.inet.0: 34 destinations, 59 routes (31 active, 0 holddown, 3 hidden)
100.0.0.1/32 (2 entries, 1 announced)
...
*BGP Preference: 170/-101
Next hop type: Router, Next hop index: 699
Address: 0x76a5cc4
Next-hop reference count: 23, key opaque handle: 0x0, non-key opaque handle: 0x0
Source: 172.18.1.1
Next hop: 172.18.1.1 via ge-0/0/0.0, selected
...
BGP Preference: 170/-101
Route Distinguisher: 192.168.0.2:101
Next hop type: Indirect, Next hop index: 0
Address: 0x76a72a4
Next-hop reference count: 26, key opaque handle: 0x0, non-key opaque handle: 0x0
Source: 192.168.0.210
Next hop type: Router, Next hop index: 0
Next hop: 10.101.1.1 via ae11.101 weight 0x1, selected
Label operation: Push 54
Label TTL action: prop-ttl
Load balance label: Label 54: Entropy label;
Label element ptr: 0x7cc8e60
Label parent element ptr: 0x0
Label element references: 5
Label element child references: 1
Label element lsp id: 0
Session Id: 0
Next hop: 10.102.1.1 via ae21.101 weight 0x4000
Label operation: Push 53
Label TTL action: prop-ttl
Load balance label: Label 53: Entropy label;
Label element ptr: 0x7cc9810
Label parent element ptr: 0x0
Label element references: 2
Label element child references: 0
Label element lsp id: 0
Session Id: 0
Protocol next hop: 192.168.101.2
Label operation: Push 16
Label TTL action: prop-ttl
Load balance label: Label 16: None;
Composite next hop: 0x6f7ec34 1048607 INH Session ID: 387
Indirect next hop: 0x7162ef4 1048604 INH Session ID: 387
State: <Secondary NotBest Int Ext ProtectionCand>
Inactive reason: Not Best in its group - Interior > Exterior > Exterior via Interior
In Junos, there is a method to protect against directly connected PE1-CE1 interface failure. Both available routes in the RIB can be copied to the FIB by adding the following command:
Figure 6: Fast reroute via two backup paths towards CE1
set routing-instances VRF-A protocols bgp group ebgp family inet unicast protection
After adding the command above, Junos marks '@' for routes used in the routing table and also displays what is installed in the forwarding table marked with '#', while maintaining the same logic of weight for the best and backup routes.
mnayman@PE1> show route 100.0.0.1/32
VRF-A.inet.0: 33 destinations, 68 routes (30 active, 0 holddown, 3 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both
100.0.0.1/32 @[BGP/170] 00:04:50, localpref 100
AS path: 101 I, validation-state: unverified
> to 172.18.1.1 via ge-0/0/0.0
[BGP/170] 00:00:09, localpref 100, from 192.168.0.210
AS path: 101 I, validation-state: unverified
> to 10.101.1.1 via ae11.101, Push 16, Push 54(top)
to 10.102.1.1 via ae21.101, Push 16, Push 53(top)
#[Multipath/255] 00:04:50
> to 172.18.1.1 via ge-0/0/0.0
to 10.101.1.1 via ae11.101, Push 16, Push 54(top)
to 10.102.1.1 via ae21.101, Push 16, Push 53(top)
Show command with extensive and expanded-nh option:
When examining the Backup route in detail, the next-hop is a composite to 192.168.101.2, which is PE2 announcing the same prefix 100.0.0.1/32 with VPN label 16 associated.
Reachability to PE2 (192.168.101.2) is unilist with two unicast via directly connected interfaces:
Label 54 via ae11.
Label 53 via ae21.
Note: Label 54 will be the best route towards PE2.
mnayman@PE1> show route 192.168.101.2 extensive | match "Destination|Index: [1-9]|weight|Push"
inet.0: 36 destinations, 49 routes (36 active, 0 holddown, 0 hidden)
KRT in-kernel 192.168.101.2/32 -> {list:Push 54, Push 53}
Next hop: 10.101.1.1 via ae11.101 weight 0x1, selected
Label operation: Push 54
Next hop: 10.102.1.1 via ae21.101 weight 0x4000
Label operation: Push 53
Next hop type: Router, Next hop index: 723
Label operation: Push 53
Figure 7: Hierarchical Routing Structure with protection feature.
Protection is also useful for multiple eBGP links to a CE. In this example, protection will be configured under the VRF with both routes active, one as primary and the other as backup.
Figure 8: Protection within a VRF for two directly connected eBGP sessions to the same CE
There are two eBGP sessions over two links to CE1.
mnayman@PE1> show route 100.0.0.1/32
VRF-A.inet.0: 32 destinations, 81 routes (32 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both
100.0.0.1/32 @[BGP/170] 00:04:02, localpref 100
AS path: 101 I, validation-state: unverified
> to 172.18.1.1 via ge-0/0/0.0
[BGP/170] 00:04:02, localpref 100
AS path: 101 I, validation-state: unverified
> to 172.18.1.3 via ge-0/0/2.0
#[Multipath/255] 00:04:02
> to 172.18.1.1 via ge-0/0/0.0
to 172.18.1.3 via ge-0/0/2.0
mnayman@PE1> request pfe execute target fpc0 command "show nhdb id 1048589 extensive"
SENT: Ukern command: show nhdb id 1048589 extensive
ID Type Interface Next Hop Addr Protocol Encap MTU Flags PFE internal Flags
----- -------- ------------- --------------- ---------- ------------ ---- ------------------ ------------------
1048589 Unilist ge-0/0/0.0 - IPv4 Ethernet 0 0x0000000000000000 0x0000000000000000
BFD Session Id: 0
Load balance flags:
per member accounting : OFF
inline jflow : OFF
random mode : OFF
rotate hash : OFF
adaptive lb : OFF
Unilist: 0
Selector ( ):
ID:38(1), Ref:1, Type:1 (Compact), subtype:0, Symmetric-LB: Off, Target_id:0
Key:FRR:Y, Balances:N, Locality:N/unicast, Type:Unicast, Size:2, flags:0x14, dist-mode-default
Weight Info (Selector's view): Current Weight = 1, Target_id = 0
Idx Balance Weight Orig-Weight Ifl Session Install
----- ------- ------- ----------- ------ ------- -------
0 ** 1 1 342 323 Yes
1 ** 16384 16384 344 322 No
Unilist: 1
Unilist Table (2 entries): List flags 0x0000000000000000
Unilist core-facing-index: 0
Underlying ifl-index : 0
HFRR force ifl selector : NO
ECMP : NO
688 Unicast ge-0/0/0.0 172.18.1.1 IPv4 Ethernet 1500 0x0000000000000000 0x0000000000000000
686 Unicast ge-0/0/2.0 172.18.1.3 IPv4 Ethernet 1500 0x0000000000000000 0x0000000000000000
Weight Info: Current Weight = 1
ID Balance Orig-Balance Weight Orig-Weight State Install Flags
----- ------- --------- ------ ----------- -------- ----------- -----
688 0 0 1 1 Active Installed 0x00
686 0 0 16384 16384 Standby Installed 0x00
Routing-table id: 0
preserve-nexthop-hierarchy - BGP Transport Layer Protection¶
Introduced in Junos 20.2, Preserve-nexthop-hierarchy is helping for accurate resolution in the forwarding plane. It maintains the hierarchical structure and weight of multiple next-hops when copying routes from the control plane to the forwarding plane. This ensures that the forwarding plane accurately reflects the control plane's routing decisions.
When the next-hop is directly connected, the PFE can automatically perform local repair, similar to interface down-based Fast Reroute (FRR), with the RPD acting as a resolver.
In some scenarios, a route to a destination may involve multiple levels of indirect next hops, such as a static route with a next-hop pointing to an indirect next hop. For example, a static route to 7.7.7.7/32 with a next hop of 192.168.224.1, advertised by 192.168.101.3. However, this chain is not installed in the routing or forwarding table.
When a remote indirect next hop becomes unreachable and is removed from the routing table, it takes longer for the VPN prefix to recover and install a new label stack in the routing and forwarding plane.
With 'preserve-nexthop-hierarchy' enabled, when the BGP route resolves over the static route, the indirect nexthops of the static route are preserved and converted into a different indirect nexthop called 'FRR-Indirect nexthop.'
Inheritance of 'flex-encap' information is supported for Flexible tunnel interfaces.
Note: Support for SR-MPLS and SRv6 is available in Junos 21.3R1.
set routing-options resolution preserve-nexthop-hierarchy
After the change, FRR for the indirect next hop is added with VPN label 16. It will be reachable via label 55 as the primary path through ae11.101 or label 54 as the backup path through ae21.101.
Figure 9: Hierarchical Routing Structure on PE1 for a destination with two INH
The 'Preserve-nexthop-hierarchy' feature and 'composite-nexthops' are necessary to support higher scales and features like BGP PIC for routes resolved over BGP CT.
multipath-resolve - Recursive Indirect Next Hop Resolution with Multipath/PIC.¶
This feature was introduced in Junos 17.3R1 to further enhance the resolver's capabilities. It enables the RPD recursive resolution feature in Junos to work with multipath/PIC, allowing for the resolution of routes over multiple IBGP routes. This feature leverages all feasible paths in the resolution hierarchy as next hops in the FIB.
This is particularly useful in scenarios where a single PE is announcing a BGP service prefix, and that PE loopback is announced by two BGP-LU peers. In other words, next hop resolution is required from the resolver to resolve BGP service route over BGP-LU routes which also have Indirect Next Hop.
The "multipath-resolve" configuration serves as a resolver import policy applied to all RIBs containing routes that require the capability for recursive resolution with Multipath/PIC.
This feature is used along with "preserve-nexthop-hierarchy", which preserves nexthop hierarchy in the FIB, to provide BGP PIC for service-routes resolving over BGP LU.
When these features are enabled, the next hop of the route being resolved over a multipath route includes all paths of the multipath route, with appropriate "weight" indicating whether a nexthop member is active or backup in FIB. This ensures that the route is properly resolved, and all available paths are considered for forwarding traffic.
To clarify:
Resolution is a mechanism through which protocols determine the immediate next-hop router through which they can reach a neighbor that is multiple hops away.
Resolver is a component in RPD that aids in the resolution of service routes.
set policy-options policy-statement LU-RESOLVER then multipath-resolve
set routing-options resolution rib inet.0 import LU-RESOLVER
set routing-options resolution rib inet.3 import LU-RESOLVER
set routing-options resolution rib bgp.l3vpn.0 import LU-RESOLVER
set routing-options resolution rib bgp.l3vpn-inet6.0 import LU-RESOLVER
multipath list-nexthop - Support for Color Transport¶
Introduced in Junos 21.1R1.
Lastly, in the ever-evolving landscape of Junos, is the 'list-nexthop' feature.
Provide multipath and protection for service routes that resolving over colored transport routes. This functionality was introduced along with BGP-CT and is leveraged for SRv6.
To create a List-NH (List Next Hop), BGP Multipath should be enabled. This configuration preserves the Transport-class of the component Indirect-Nexthops within the Multipath formation. This feature proves useful for maintaining Transport-class information when using Multipath in BGP routing.
'List-nexthop' enhances route resolution for BGP routes that utilize List-NH, optimizes re-resolution within recursive resolution for multipath routes, and implements a decoupled resolver to enhance convergence and boost performance.
This feature also applies to service routes with homogeneous next hops, which can be either eBGP or iBGP.
For SRv6, the 'list-nexthop' and 'preserve-nexthop-hierarchy' features are required for proper next-hop resolution of services using SRv6 and, consequently, IPv6 next hops.
In conclusion, although BGP is a capable protocol, routers and networks should not rely solely on it as the exclusive option. Network engineers should have the freedom to choose the most suitable routing protocols for constructing their networks.
Alternative solutions are available in the form of IGP protocols like OSPF and IS-IS. The 'BGP-Free Core' concept has been embraced by many service provider networks and continues serve as the foundation for core networks. IGP protocols are intentionally designed for internal networks, incorporating mechanisms like link metrics, bandwidth, delay, and cost to determine the optimal path within a network.
Thanks to Ravi M R, Navaneetha Krishnan V and the team for developing the feature set in Junos OS. And thanks to Kaliraj Vairavakkalai, Ravi M R for review and consultation during this blog.
