The second part of the “VXLAN EVPN DCI” project regards underlay configurations in terms of:
- VPC domain
- Ling-aggregation
- IGP routing
VPC domain is a Cisco feature on the Nexus switches that provides the ability to configure a Port-Channel across multiple switches (i.e. VPC peers). The VPC peers, configured into a VPC domain, act as a single switch from data-plane perspective but operates independetly at control-plane.
Recalling the lab topology, i need to configure 2 different VPC domain (i.e. 100 and 200), one domain at each DC site.
Except for domain id, the configuration is quite similar between the border gateway:
e.g. SiteABGW1
feature vpc
feature lacp
vrf context KeepAlive
interface Ethernet1/5
description "VPC Keepalive"
no switchport
vrf member KeepAlive
ip address 169.254.0.1/30
no shutdown
vpc domain 100
peer-switch
role priority 16334
peer-keepalive destination 169.254.0.2 source 169.254.0.1 vrf KeepAlive
delay restore 150
peer-gateway
layer3 peer-router
auto-recovery
auto-recovery reload-delay 360
ip arp synchronize
interface Ethernet1/6-7
description "VPC PeerLink"
switchport mode trunk
channel-group 6 mode active
interface port-channel6
description "VPC PeerLink"
switchport mode trunk
spanning-tree port type network
vpc peer-link
First of all, i need to activate the necessary feature for this configuration purpose (vpc and lacp). Then i configure the VPC keepalive port, i prefer to configure it as a routed port in a different vrf in order to separate this kind of funcionality and maintain the global RIB clean.
The second step is to define the VPC domain where i can configure many parameters to optimize the peers behavior. In this case i configure the following parameters:
- peer-switch: allows a pair of vPC peers to appear as a single STP root in the Layer 2 topology.
- role priority: allow to predetermine the vpc primary peer.
- peer-keepalive destination: define the destination ip address of the remote vpc peer in order to check its health.
- delay restore: this is the timer that prevents the VPC on a failed switch coming back into service until it can actually pass traffic.
- peer-gateway: allows a vPC switch to act as the active gateway for packets that are addressed to the router MAC address of the vPC peer. In our case the border gateway will not act as a gateway but this command is a prerequisite to enable “layer3 peer-router”.
- layer3 peer-router: this command tells the Nexus to not decrement the TTL so you can achieve routing adjacency over vpc peer-link.
- auto-recovery: provide a backup mechanism in case of vPC peer-link failure followed by vPC primary peer device failure.
- auto-recovery reload-delay: if both peers reload, allows the unique alive vPC peer device to assume the vPC primary role and bring up all local vPCs ports after the expiration of the delay timer.
- ip arp synchronize: improves convergence time for Layer 3 flows performing an ARP bulk synchronization.
Finally i need to configure a port-channel and define it as “vpc peer-link” (Po6).
The final result should be like the output below:
SiteABGW1# sh vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 100
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 2
Peer Gateway : Enabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled, timer is off.(timeout = 360s)
Delay-restore status : Timer is on.(timeout = 150s, 138s left)
Delay-restore SVI status : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router : Enabled
Virtual-peerlink mode : Disabled
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ -------------------------------------------------
1 Po6 up 1,10,20,30,999
The second part of the underlay configuration related to link-aggregation. Link-aggregation, using the standard protocol LACP, allow me to aggregate different physical link into a logical link called port-channel.
Here the simple configuration of port-channel 3 using the Eth1/3 as physical interface:
e.g. SiteABGW1
interface Ethernet1/3
description TO_SiteA-1
lacp rate fast
channel-group 3 mode active
interface port-channel3
description TO_SiteA-1
switchport mode trunk
switchport trunk allowed vlan 10,30
vpc 3
As you can see from the configuration, on the physical port i enable lacp fast rate in order to enable fast failover. On the logical interface (Po3) i activate a distributed port-channel with the command “vpc 3”. This command allow me to create a port-channel using physical ports on different switches (vpc peers):
SiteABGW1# sh vpc 3
vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
3 Po3 up success success 10,30
The final part of the underlay configuration is IGP routing, in this case OSPF. Firstly we need to define a couple of Loopback interface, one for the underlay routing (Lo0) and one for the VTEP (VXLAN Tunnel Endpoint - Lo1) address, and advertise them into the OSPF process. In this lab i also use Eth1/1-2 as an ip unnumbered interfaces binded to Lo0 to reduce the number of subnet/ip addresses in use. I also create a new vlan 999 for the vpc peers adjacency:
e.g. SiteABGW1
feature ospf
feature bfd
feature interface-vlan
bfd interval 250 min_rx 250 multiplier 4
router ospf UNDERLAY
bfd
interface Ethernet1/1-2
description TO_SiteB
no switchport
mtu 9216
medium p2p
ip unnumbered loopback0
ip router ospf UNDERLAY area 0.0.0.0
no shutdown
interface loopback0
description VXLAN-UNDERLAY
ip address 172.16.0.1/32
ip router ospf UNDERLAY area 0.0.0.0
interface loopback1
description VTEP addresses
ip address 172.16.1.1/32
ip address 172.16.1.100/32 secondary
ip router ospf UNDERLAY area 0.0.0.0
interface Vlan999
no shutdown
no ip redirects
ip address 192.168.99.1/30
no ipv6 redirects
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
The final result should be:
SiteABGW1# sh ip ospf interface brief
OSPF Process ID UNDERLAY VRF default
Total number of interface: 7
Interface ID Area Cost State Neighbors Status
Vlan999 3 0.0.0.0 40 P2P 1 up
Lo0 1 0.0.0.0 1 LOOPBACK 0 up
Lo1 2 0.0.0.0 1 LOOPBACK 0 up
Eth1/1 7 0.0.0.0 40 P2P 1 up
Eth1/2 6 0.0.0.0 40 P2P 1 up
SiteABGW1# sh ip ospf neighbors
OSPF Process ID UNDERLAY VRF default
Total number of neighbors: 3
Neighbor ID Pri State Up Time Address Interface
172.16.0.2 1 FULL/ - 00:30:06 192.168.99.2 Vlan999
172.16.0.3 1 FULL/ - 00:02:07 172.16.0.3 Eth1/1
172.16.0.4 1 FULL/ - 00:01:48 172.16.0.4 Eth1/2
SiteABGW1# sh ip route ospf-UNDERLAY
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
172.16.0.2/32, ubest/mbest: 1/0
*via 192.168.99.2, Vlan999, [110/41], 00:08:58, ospf-UNDERLAY, intra
172.16.0.3/32, ubest/mbest: 1/0
*via 172.16.0.3, Eth1/1, [110/41], 00:09:10, ospf-UNDERLAY, intra
via 172.16.0.3, Eth1/1, [250/0], 00:32:11, am
172.16.0.4/32, ubest/mbest: 1/0
*via 172.16.0.4, Eth1/2, [110/41], 00:08:59, ospf-UNDERLAY, intra
via 172.16.0.4, Eth1/2, [250/0], 00:31:53, am
172.16.1.2/32, ubest/mbest: 1/0
*via 192.168.99.2, Vlan999, [110/41], 00:08:58, ospf-UNDERLAY, intra
172.16.1.3/32, ubest/mbest: 1/0
*via 172.16.0.3, Eth1/1, [110/41], 00:09:10, ospf-UNDERLAY, intra
172.16.1.4/32, ubest/mbest: 1/0
*via 172.16.0.4, Eth1/2, [110/41], 00:08:59, ospf-UNDERLAY, intra
172.16.1.200/32, ubest/mbest: 2/0
*via 172.16.0.3, Eth1/1, [110/41], 00:09:10, ospf-UNDERLAY, intra
*via 172.16.0.4, Eth1/2, [110/41], 00:08:59, ospf-UNDERLAY, intra
192.168.99.4/30, ubest/mbest: 2/0
*via 172.16.0.3, Eth1/1, [110/80], 00:09:10, ospf-UNDERLAY, intra
*via 172.16.0.4, Eth1/2, [110/80], 00:08:59, ospf-UNDERLAY, intra
N.B. BFD feature is not currently supported on 9000v version, you can configure it but all the sessions will remain in down state!
The final underlay L3 diagram should looks like this:
In the next post i will focus on overlay configuration.