Genie's Tech Blog

Where knowledge has no dimensions

BGP-based VPLS Autodiscovery

Hello Everyone,

I am back with yet another post on one of the topics which caught my interest on - VPLS Autodiscovery using BGP. Before we start digging in to it, lets first start by understanding what VPLS is?

What is VPLS?

Virtual - Multiple VPLSs can be offered over common packet switched network.

Private - CE devices that belong to different VPLSs cannot interact

LAN Service - CE devices that belong to given VPLS instance V, can interact through the SP network as if they were connected by a LAN.

VPLS is an architecture which allows MPLS networks to provide Multipoint Ethernet LAN Service also referred to as Transparent LAN Service (TLS). VPLS enables geographically separate LAN segments to be interconnected as a single bridged domain over a packet switched network, such as IP, MPLS or hybrid of both. VPLS is specifically oriented to Service Providers, as a service. Customer is wholly unaware of the details of emulation and customer appears to have a single broadcast domain. In a VPLS deployment model, the service provider backbone network acts as a logic bridge. The topology and signaling of the backbone are transparent to customer LAN segments that are interconnected as if they are attached to a series of LAN switches. In a service provider network, the topology for a particular VPLS domain is typically set up as one of the following topology models: hub-and-spoke, full-mesh, and hierarchical. Below is a sample topology on how VPLS setup looks:

 

In a hub-and-spoke model, the PE router that acts as the hub establishes a point-to-multipoint forwarding relationship with all PE routers at the spoke sites.  An Ethernet or VLAN packet received from the customer network on the hub PE can be forwarded to one or more emulated VCs.  On the other hand, the PE routers that act as the spoke establish a point-to-point connection to the PE at the hub site. An Ethernet or VLAN packet received from the customer network on the spoke PE can only be forwarded to the VC destined to the hub site.  

Whether to create a point-to-multipoint or point-to-point connection is purely a local forwarding matter determined by the configuration on the PE in question. The VC types signaled through the signaling session on all PE routers are identical. From the configuration point of view, the only difference is that there is only one peering PE router on the spoke PE router, while there is more than one peering PE router on the hub PE router. Such a configuration sample will be provided in the end user interface section. The hub-and-spoke model is also useful in the case of connecting legacy point-to-point PE routers at the spoke sites to a VPLS-capable PE router at the hub site.  More details are described in the backward compatibility section. By default, layer-2 split horizon is enabled in the data path and no layer-2 traffic is allowed to exchange between spoke sites directly. This does not prevent the spoke sites having layer-3 connectivity, however.

In a full-mesh model, each PE router creates a multipoint-to-multipoint forwarding relationship with all other PE routers in the VPLS domain by the way of layer-2 Virtual Forwarding Instance (VFI), which will be further explained in the later sections. An Ethernet or VLAN packet received from the customer network can be forwarded to one or more local interfaces and/or emulated VCs in the VPLS domain. To avoid broadcasted packets looping around in the network, no packet received from an emulated VC can be forwarded to any emulated VC of the VPLS domain on a PE router. That is, the layer-2 split horizon should always be enabled as default in a full-mesh network.

A hierarchical VPLS model comprises both hub-and-spoke and full-mesh networks. Although it probably has the best performance to management cost ratio, its signaling procedures and data forwarding behaviors are far more complex than the pure hub-and-spoke or full-mesh model. In the current phase of development, only the basic form of hierarchical VPLS that requires no additional signaling procedure is supported by the way of a configurable parameter that instructs the data plane to enable or disable layer-2 split horizon on per neighbor basis.

Two types of VPLS services will be available to VPLS customers: Transparent LAN Service (TLS) and Ethernet Virtual Connection Service (EVCS). 

With TLS, the PE router forwards all Ethernet packets received from the customer-facing interface, including tagged, untagged, and BPDU, to a local Ethernet interface or an emulated VC if the destination MAC address is found in the L2 forwarding table, or all other local Ethernet interfaces and emulated VCs belonging to the same VPLS domain if the destination MAC address is a multicast/broadcast address or not found in the L2 forwarding table.

With EVCS, the PE router forwards all Ethernet packets with a particular VLAN tag(s) received from the customer-facing interface, excluding BPDU, to a local Ethernet interface or an emulated VC if the destination MAC address is found in the L2 forwarding table, or all other local Ethernet interfaces and emulated VCs belonging to the same VPLS domain if the destination MAC address is a multicast/broadcast address or not found in the L2 forwarding table. The demultiplexing VLAN tag(s) used to identify a VPLS domain is removed prior to forwarding the packet to the outgoing Ethernet interfaces or emulated VCs because they only have local significance.

The signaling procedures of setting up a VC for VPLS is quite similar to setting up a VC for point-to-point Ethernet connection. The first step involves creating a signaling session between each pair of peering PE routers. The participating PE routers of a given VPLS domain are determined through either manual configuration or auto-discovery. In this post we are specifically going to talk about BGP-based VPLS autodiscovery.
 
BGP-based VPLS Autodiscovery has been described in RFC 4761. BGP-based VPLS Autodiscovery eliminates the need to manually provision a VPLS neighbor. After a PE configures itself to be a member of a particular VPLS, information needed to setup connections to remote PEs in the same VPLS is distributed by a discovery process. When the discovery process is complete, each PE member of the VPLS will have the information needed to setup VPLS Pseudowires (PWs) to form the full mesh of PWs needed for the VPLS. Please note that VPLS Autodiscovery uses FEC 129 to convey endpoint information. FEC 129 will be used only for auto-discovered VPLS PWs. Manually configured PWs will continue to use FEC128, VPLS and VPWS. Auto-discovery proceeds by having each PE distribute, via BGP, the NLRI for each of its VFIs with itself as the BGP next hop, and with the appropriate RT for each such NLRI. Typically, each PE would be a client of a small set of BGP route reflectors, which would redistribute this information to the other clients. If a PE has a VFI with a particular RT, it can then import all the NLRI which have that same RT, and from the BGP next hop attribute of these NLRI it will learn the IP addresses of the other PE routers which have VFIs with the same RT. Local VFI configuration triggers the advertisement of local endpoints. The endpoint data is put into the NLRI and given to BGP for distribution to remote PEs. The exported and imported RT values associated with this NLRI are also given to BGP. The remote PE(s) receive the NLRI with NH(s) from BGP. BGP gives this information to L2VPN. The NLRI is used to uniquely identify the VFI, and the VPLS-ID is used to identify the VPLS domain. The below diagram depicts the high level overview of how BGP-based VPLS autodiscovery works.

The following information will be supplied by BGP to L2VPN:

- Filtered NLRI (from remote PEs): RT Filtering.
- One or more NHs per NLRI
- A source IP address per NH address
- Matching import RT list per NLRI
- AS number upon request
- AS number change or BGP restart indication: This is used to inform L2VPN of an AS number change.  This enables L2VPN to cache the AS number for performance reasons.
- BGP Refresh API: Request sent to BGP from L2VPN requesting NLRI updates. The scope of the updates may be limited by NLRI and/or RT. BGP would then respond by giving L2VPN a refresh of all associated NLRI that has been previously announced by remote PEs. This is called a refresh because BGP will provide the requested NLRI, even though BGP may have already given L2VPN the same NLRI. At this time, it appears that this API will not be needed.

The following information will be supplied by L2VPN to BGP:

- NLRI (newly created)
- One or more RTs per NLRI with import/export flags for each RT
- All local NLRI upon request

We will now see how BGP-based VPLS autodiscovery works with the help of an example. Lets consider the topology as shown in first figure. Lets have a P (Core) router as a part of the MPLS network. In this example, we have two PE routers as Cisco 7600 router and one of the PE as ASR1000 router. So we shall now see how BGP-based VPLS Autodiscovery works on different platforms and how its configured. Below is the hardware and software description for the 7600 and the ASR1000 router used in this setup.

Cisco 7600 Platform: 

RSP: RSP720-3C-GE
Core and CE facing Linecard: 76-ES+XC-20G3CXL /  76-ES+XC-20G3C   (Both the PE routers is having either of the Linecards)
Software: c7600rsp72043-advipservicesk9-mz.122-33.SRE4.bin

Cisco ASR1004 Platform: 

Route Processor: ASR1000-RP2
ESP: ASR1000-ESP10
Core and CE facing Linecard: ASR1000-SIP10    SPA: SPA-5X1GE-V2
Software: asr1000rp2-adventerprisek9.03.06.02.S.152-2.S2.bin

Lets now have a look at the complete configuration of the setup we have here:

Configuration on PE1: Cisco 7606
================================
mpls label protocol ldp
mpls label range 201 300
!
l2 vfi vpls1 autodiscovery 
 vpn id 1001
!         
l2 vfi vpls2 autodiscovery 
 vpn id 1000
!
interface Loopback0
 ip address 59.0.0.1 255.255.255.255
!
interface GigabitEthernet4/1
 ip address 12.12.12.1 255.255.255.0
 speed 1000
 mpls ip
!
interface GigabitEthernet4/2
 bandwidth 1000000
 no ip address
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 speed 1000
 service instance 1000 ethernet
  encapsulation dot1q 1000
  rewrite ingress tag pop 1 symmetric
  l2protocol forward
  bridge-domain 1000
 ! 
  service instance 1001 ethernet
  encapsulation dot1q 1001
  rewrite ingress tag pop 1 symmetric
  l2protocol forward
  bridge-domain 1001
 !
!
interface Vlan1000
 no ip address
 xconnect vfi vpls2
!
interface Vlan1001
 no ip address
 xconnect vfi vpls1
!
router ospf 100
 router-id 59.0.0.1
 log-adjacency-changes
 network 12.12.12.1 0.0.0.0 area 0
 network 59.0.0.1 0.0.0.0 area 0
!
router bgp 100
 bgp router-id 59.0.0.1
 no bgp default ipv4-unicast
 bgp log-neighbor-changes
 neighbor 59.0.0.3 remote-as 100
 neighbor 59.0.0.3 update-source Loopback0
 neighbor 59.0.0.4 remote-as 100
 neighbor 59.0.0.4 update-source Loopback0
 !
 address-family ipv4
  no synchronization
  neighbor 59.0.0.3 activate
  neighbor 59.0.0.3 send-community
  neighbor 59.0.0.3 next-hop-self
  neighbor 59.0.0.4 activate
  neighbor 59.0.0.4 send-community
  neighbor 59.0.0.4 next-hop-self
  no auto-summary
 exit-address-family
 !
  address-family l2vpn vpls
  neighbor 59.0.0.3 activate
  neighbor 59.0.0.3 send-community extended
  neighbor 59.0.0.4 activate
  neighbor 59.0.0.4 send-community extended
 exit-address-family
!
mpls ldp router-id Loopback0 force
!

Configuration on PE2: Cisco 7606
================================
mpls label protocol ldp
mpls label range 301 400
!
l2 vfi vpls1 autodiscovery 
 vpn id 1001
!
l2 vfi vpls2 autodiscovery 
 vpn id 1000
!
interface Loopback0
 ip address 59.0.0.3 255.255.255.255
!
interface GigabitEthernet4/1
 ip address 23.23.23.3 255.255.255.0
 speed 1000
 mpls ip
!
interface GigabitEthernet4/2
 bandwidth 1000000
 no ip address
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 speed 1000
 service instance 1000 ethernet
  encapsulation dot1q 1000
  rewrite ingress tag pop 1 symmetric
  l2protocol forward
  bridge-domain 1000
 !
 service instance 1001 ethernet
  encapsulation dot1q 1001
  rewrite ingress tag pop 1 symmetric
  l2protocol forward
  bridge-domain 1001
 !
!
interface Vlan1000
 no ip address
 xconnect vfi vpls2
!
interface Vlan1001
 no ip address
 xconnect vfi vpls1
!
router ospf 100
 router-id 59.0.0.3
 log-adjacency-changes
 network 23.23.23.3 0.0.0.0 area 0
 network 59.0.0.3 0.0.0.0 area 0
!
router bgp 100
 bgp router-id 59.0.0.3
 no bgp default ipv4-unicast
 bgp log-neighbor-changes
 neighbor 59.0.0.1 remote-as 100
 neighbor 59.0.0.1 update-source Loopback0
 neighbor 59.0.0.4 remote-as 100
 neighbor 59.0.0.4 update-source Loopback0
 ! 
  address-family ipv4
  no synchronization
  neighbor 59.0.0.1 activate
  neighbor 59.0.0.1 send-community
  neighbor 59.0.0.1 next-hop-self
  neighbor 59.0.0.4 activate
  neighbor 59.0.0.4 send-community
  neighbor 59.0.0.4 next-hop-self
  no auto-summary
 exit-address-family
 !        
 address-family l2vpn vpls
  neighbor 59.0.0.1 activate
  neighbor 59.0.0.1 send-community both
  neighbor 59.0.0.4 activate
  neighbor 59.0.0.4 send-community extended
 exit-address-family
! 
mpls ldp router-id Loopback0 force
!

Configuration on PE3: - Cisco ASR1004
======================================
mpls label range 401 500
mpls label protocol ldp
!
l2 vfi vpls1 autodiscovery 
 vpn id 1001
 bridge-domain 1001
!
l2 vfi vpls2 autodiscovery 
 vpn id 1000
 bridge-domain 1000
!
interface Loopback0
 ip address 59.0.0.4 255.255.255.255
!
interface GigabitEthernet0/0/0
 ip address 24.24.24.4 255.255.255.0
 negotiation auto
 mpls ip
 cdp enable
!
interface GigabitEthernet0/0/1
 bandwidth 1000000
 no ip address
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 negotiation auto
 service instance 1000 ethernet
  encapsulation dot1q 1000
  rewrite ingress tag pop 1 symmetric
  bridge-domain 1000
 !        
 service instance 1001 ethernet
  encapsulation dot1q 1001
  rewrite ingress tag pop 1 symmetric
  bridge-domain 1001
 !
!
router ospf 100
 router-id 59.0.0.4
 network 24.24.24.4 0.0.0.0 area 0
 network 59.0.0.4 0.0.0.0 area 0
!         
router bgp 100
 bgp router-id 59.0.0.4
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 59.0.0.1 remote-as 100
 neighbor 59.0.0.1 update-source Loopback0
 neighbor 59.0.0.3 remote-as 100
 neighbor 59.0.0.3 update-source Loopback0
 !        
 address-family ipv4
  neighbor 59.0.0.1 activate
  neighbor 59.0.0.1 send-community
  neighbor 59.0.0.1 next-hop-self
  neighbor 59.0.0.3 activate
  neighbor 59.0.0.3 send-community
  neighbor 59.0.0.3 next-hop-self
 exit-address-family
 ! 
  address-family l2vpn vpls
  neighbor 59.0.0.1 activate
  neighbor 59.0.0.1 send-community extended
  neighbor 59.0.0.3 activate
  neighbor 59.0.0.3 send-community extended
 exit-address-family
!  
mpls ldp router-id Loopback0 force
!

Configuration on P: Cisco 7200
==============================
mpls label range 101 200
mpls label protocol ldp
!
interface Loopback0
 ip address 59.0.0.2 255.255.255.255
!
interface GigabitEthernet0/1
 ip address 23.23.23.2 255.255.255.0
 media-type rj45
 speed auto
 duplex auto
 negotiation auto
 mpls ip
!
interface FastEthernet0/2
 no ip address
 shutdown
 speed auto
 duplex auto
!
interface GigabitEthernet0/2
 ip address 12.12.12.2 255.255.255.0
 media-type rj45
 speed auto
 duplex auto
 negotiation auto
 mpls ip  
!         
interface GigabitEthernet0/3
 ip address 24.24.24.2 255.255.255.0
 media-type rj45
 speed auto
 duplex auto
 negotiation auto
 mpls ip  
!         
router ospf 100
 router-id 59.0.0.2
 log-adjacency-changes
 network 12.12.12.2 0.0.0.0 area 0
 network 23.23.23.2 0.0.0.0 area 0
 network 24.24.24.2 0.0.0.0 area 0
 network 59.0.0.2 0.0.0.0 area 0
!         
mpls ldp router-id Loopback0 force
!

Configuration on CE1: Cisco 3750
=================================
interface GigabitEthernet5/0/1
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface Vlan1000
 ip address 192.168.2.1 255.255.255.0
!
interface Vlan1001
 ip address 192.168.1.1 255.255.255.0
!

Configuration on CE2: Cisco 3750
=================================
interface GigabitEthernet1/0/1
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface Vlan1000
 ip address 192.168.2.2 255.255.255.0
!
interface Vlan1001
 ip address 192.168.1.2 255.255.255.0
!

Configuration on CE3: Cisco 3750
=================================
interface GigabitEthernet1/0/1
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface Vlan1000
 ip address 192.168.2.3 255.255.255.0
!
interface Vlan1001
 ip address 192.168.1.3 255.255.255.0
!

From the above config you can see that the P router i.e. the 7200 router is just being used for the IGP connectivity and maintaining the MPLS connectivity. We can use this router also as the Route-Reflector (RR) for the the setup. So instead of having full mesh connectivity, we can have the all the PE routers peering with the RR. Please remember that "l2vpn vpls" BGP address-family is required to enable BGP-based VPLS. A VPLS BGP NLRI has the following information elements: a VE ID, a VE Block Offset, a VE Block Size, and a label base. The format of the VPLS NLRI is given below. The AFI is the L2VPN AFI (25), and the SAFI is the VPLS SAFI (65). The Length field is in octets.

 

The "Layer2 Info Extended Community", is used to signal control information about the pseudowires to be setup for a given VPLS. The extended community value (0x800A) is to be allocated by IANA. 

 

We often have a lot of confusion on the control plane aspect. Its important to note that no MAC addresses are exchanged via BGP. All MAC address learning and aging is done in the data plane individually by each PE. The only task of BGP VPLS message exchange is auto-discovery and label exchange. Thus BGP processing for VPLS occurs when a) A PE joins or leaves a VPLS or b) A failure occurs in the network, bringing down a PE-PE tunnel or a PE-CE link.

Some important information from RFC regarding VPLS Autodiscovery:

VPLS Autodiscovery requires the LDP signaled PW to use the generalized FEC, also referred to as FEC 129 [PWE3-CONTROL] when communicating with the peer PE [L2VPN-SIG]. It is also possible for BGP to advertise a next-hop address that is different from the LDP router-id of the peer PE.

As described in [L2VPN-SIG] the 4-byte PW id is not sufficient to signal auto-discovered PWs for VPLS. FEC 129 is required and will be encoded as follows when sending VC-specific LDP messages:

- AGI =VPLS-ID from BGP NLRI
- TAII = Peer PE L2VPN router-id from BGP NLRI
- SAII = Local L2VPN router-id

AGI and AII types are type 0x01 as specified in [PWE3-IANA]

It is possible to use FEC 129 to signal manually provisioned PWs. The current CLI for VPLS and VPWS forces the same VC-id on the local and remote PEs, so the FEC 129 encoding would be: AGI=none, TAII = VC-id, SAII = VC-id.

Hope this information helps understand what information is exchanged between the PE routers when VPLS is setup. Lets now have a look at the few outputs:

Output on PE1:
==========
PE1#sh mpls l2 vc 

Local intf     Local circuit              Dest address    VC ID      Status    
-------------  -------------------------- --------------- ---------- ----------
VFI vpls2      VFI                        59.0.0.3        1000       UP        
VFI vpls1      VFI                        59.0.0.3        1001       UP        
VFI vpls2      VFI                        59.0.0.4        1000       UP        
VFI vpls1      VFI                        59.0.0.4        1001       UP        
PE1#show xconn all
Legend:    XC ST=Xconnect State  S1=Segment1 State  S2=Segment2 State
  UP=Up       DN=Down            AD=Admin Down      IA=Inactive
  SB=Standby  HS=Hot Standby     RV=Recovering      NH=No Hardware

XC ST  Segment 1                         S1 Segment 2                         S2
------+---------------------------------+--+---------------------------------+--
UP     vfi  vpls2                        UP mpls 59.0.0.3:1000                UP
UP     vfi  vpls1                        UP mpls 59.0.0.3:1001                UP
UP     vfi  vpls2                        UP mpls 59.0.0.4:1000                UP
UP     vfi  vpls1                        UP mpls 59.0.0.4:1001                UP
UP     ac   Vl1001:1001(Eth VLAN)        UP vfi  vpls1                        UP
UP     ac   Vl1000:1000(Eth VLAN)        UP vfi  vpls2                        UP
PE1#

Output on PE2:
===========
PE2#sh mpls l2 vc

Local intf     Local circuit              Dest address    VC ID      Status    
-------------  -------------------------- --------------- ---------- ----------
VFI vpls2      VFI                        59.0.0.1        1000       UP        
VFI vpls1      VFI                        59.0.0.1        1001       UP        
VFI vpls2      VFI                        59.0.0.4        1000       UP        
VFI vpls1      VFI                        59.0.0.4        1001       UP        
PE2#show xconn all
Legend:    XC ST=Xconnect State  S1=Segment1 State  S2=Segment2 State
  UP=Up       DN=Down            AD=Admin Down      IA=Inactive
  SB=Standby  HS=Hot Standby     RV=Recovering      NH=No Hardware

XC ST  Segment 1                         S1 Segment 2                         S2
------+---------------------------------+--+---------------------------------+--
UP     vfi  vpls2                        UP mpls 59.0.0.1:1000                UP
UP     vfi  vpls1                        UP mpls 59.0.0.1:1001                UP
UP     vfi  vpls2                        UP mpls 59.0.0.4:1000                UP
UP     vfi  vpls1                        UP mpls 59.0.0.4:1001                UP
UP     ac   Vl1001:1001(Eth VLAN)        UP vfi  vpls1                        UP
UP     ac   Vl1000:1000(Eth VLAN)        UP vfi  vpls2                        UP
PE2#

Output on PE3:
==========
PE3#show mpls l2 vc

Local intf     Local circuit              Dest address    VC ID      Status
-------------  -------------------------- --------------- ---------- ----------
VFI vpls2      vfi                        59.0.0.1        1000       UP        
VFI vpls1      vfi                        59.0.0.1        1001       UP        
VFI vpls2      vfi                        59.0.0.3        1000       UP        
VFI vpls1      vfi                        59.0.0.3        1001       UP        

PE3#show xconnect all
Legend:    XC ST=Xconnect State  S1=Segment1 State  S2=Segment2 State
  UP=Up       DN=Down            AD=Admin Down      IA=Inactive
  SB=Standby  HS=Hot Standby     RV=Recovering      NH=No Hardware

XC ST  Segment 1                         S1 Segment 2                         S2
------+---------------------------------+--+---------------------------------+--
UP pri  vfi vpls1                        UP mpls 59.0.0.1:1001                UP
UP pri  vfi vpls1                        UP mpls 59.0.0.3:1001                UP
UP pri  vfi vpls2                        UP mpls 59.0.0.3:1000                UP
UP pri  vfi vpls2                        UP mpls 59.0.0.1:1000                UP
UP pri   bd 1000                         UP  vfi vpls2                        UP
UP pri   bd 1001                         UP  vfi vpls1                        UP
PE3#

If we notice the output on PE3 which is the ASR1000 router, the vfi is setup on the bridge-domain as compared to the one on 7600 which are actually setup on the vlans. Lets now have a look at the detailed output of the vc's on both 7600 and asr1k router:

Output on PE1:
==========
PE1#sh mpls l2 vc 1000 det
Local interface: VFI vpls2 VFI up    <<<<<<<<<<<<
  Interworking type is Ethernet
  Destination address: 59.0.0.3, VC ID: 1000, VC status: up
    Output interface: Gi4/1, imposed label stack {102 308}    <<<<<<<<<<
    Preferred path: not configured  
    Default path: active
    Next hop: 12.12.12.2
  Create time: 1w2d, last status change time: 1w2d
  Signaling protocol: LDP, peer 59.0.0.3:0 up
    Targeted Hello: 59.0.0.1(LDP Id) -> 59.0.0.3, LDP is UP
    Status TLV support (local/remote)   : enabled/supported
      LDP route watch                   : enabled
      Label/status state machine        : established, LruRru
      Last local dataplane   status rcvd: No fault
      Last local SSS circuit status rcvd: No fault
      Last local SSS circuit status sent: No fault
      Last local  LDP TLV    status sent: No fault
      Last remote LDP TLV    status rcvd: No fault
      Last remote LDP ADJ    status rcvd: No fault
    MPLS VC labels: local 208, remote 308 <<<<<<<<<<<<<<<
    AGI: type 1, len 8, 000A 0064 0000 03E8
    Local AII: type 1, len 4, 3B00 0001 (59.0.0.1)
    Remote AII: type 1, len 4, 3B00 0003 (59.0.0.3)
    Group ID: local n/a, remote n/a
    MTU: local 1500, remote 1500    <<<<<<<<<<<<
    Remote interface description: 
  Sequencing: receive disabled, send disabled
  Control Word: On (configured: autosense)
  VC statistics:
    transit packet totals: receive 399540, send 60  <<<<<<<<<<<
    transit byte totals:   receive 28766874, send 4296
    transit packet drops:  receive 0, seq error 0, send 0

Local interface: VFI vpls2 VFI up    <<<<<<<<<<<<<<<<<
  Interworking type is Ethernet
  Destination address: 59.0.0.4, VC ID: 1000, VC status: up
    Output interface: Gi4/1, imposed label stack {103 408}
    Preferred path: not configured  
    Default path: active
    Next hop: 12.12.12.2
  Create time: 1w2d, last status change time: 1w2d
  Signaling protocol: LDP, peer 59.0.0.4:0 up
    Targeted Hello: 59.0.0.1(LDP Id) -> 59.0.0.4, LDP is UP
    Status TLV support (local/remote)   : enabled/supported
      LDP route watch                   : enabled
      Label/status state machine        : established, LruRru
      Last local dataplane   status rcvd: No fault
      Last local SSS circuit status rcvd: No fault
      Last local SSS circuit status sent: No fault
      Last local  LDP TLV    status sent: No fault
      Last remote LDP TLV    status rcvd: No fault
      Last remote LDP ADJ    status rcvd: No fault
    MPLS VC labels: local 209, remote 408     <<<<<<<<<<<<<<<
    AGI: type 1, len 8, 000A 0064 0000 03E8
    Local AII: type 1, len 4, 3B00 0001 (59.0.0.1)
    Remote AII: type 1, len 4, 3B00 0004 (59.0.0.4)
    Group ID: local n/a, remote n/a
    MTU: local 1500, remote 1500    <<<<<<<<<<<<<<<
    Remote interface description: 
  Sequencing: receive disabled, send disabled
  Control Word: On (configured: autosense)
  VC statistics:
    transit packet totals: receive 121, send 91    <<<<<<<<<<<<<<<<<
    transit byte totals:   receive 8498, send 6404
    transit packet drops:  receive 0, seq error 0, send 0

PE1#

Output from PE2:
================
PE3#sh mpls l2 vc 1000 det
Local interface: VFI vpls2 vfi up
  Interworking type is Ethernet
  Destination address: 59.0.0.1, VC ID: 1000, VC status: up
    Output interface: Gi0/0/0, imposed label stack {101 209}
    Preferred path: not configured  
    Default path: active
    Next hop: 24.24.24.2
  Create time: 1w2d, last status change time: 1w2d
  Signaling protocol: LDP, peer 59.0.0.1:0 up
    Targeted Hello: 59.0.0.4(LDP Id) -> 59.0.0.1, LDP is UP
    Status TLV support (local/remote)   : enabled/supported
      LDP route watch                   : disabled
      Label/status state machine        : established, LruRru
      Last local dataplane   status rcvd: No fault
      Last BFD dataplane     status rcvd: Not sent
      Last BFD peer monitor  status rcvd: No fault
      Last local AC  circuit status rcvd: No fault
      Last local AC  circuit status sent: No fault
      Last local LDP TLV     status sent: No fault
      Last remote LDP TLV    status rcvd: No fault
      Last remote LDP ADJ    status rcvd: No fault
    MPLS VC labels: local 408, remote 209 
    AGI: type 1, len 8, 000A 0064 0000 03E8
    Local AII: type 1, len 4, 3B00 0004 (59.0.0.4)
    Remote AII: type 1, len 4, 3B00 0001 (59.0.0.1)
    Group ID: local n/a, remote n/a
    MTU: local 1500, remote 1500
    Remote interface description: 
  Sequencing: receive disabled, send disabled
  Control Word: On (configured: autosense)
  SSO Descriptor: 59.0.0.1/1000, local label: 408
  Dataplane:
    SSM segment/switch IDs: 16402/16399 (used), PWID: 3
  VC statistics:
    transit packet totals: receive 90, send 121
    transit byte totals:   receive 5976, send 10676
    transit packet drops:  receive 1, seq error 0, send 0

Local interface: VFI vpls2 vfi up
  Interworking type is Ethernet
  Destination address: 59.0.0.3, VC ID: 1000, VC status: up
    Output interface: Gi0/0/0, imposed label stack {102 309}
    Preferred path: not configured  
    Default path: active
    Next hop: 24.24.24.2
  Create time: 1w2d, last status change time: 1w2d
  Signaling protocol: LDP, peer 59.0.0.3:0 up
    Targeted Hello: 59.0.0.4(LDP Id) -> 59.0.0.3, LDP is UP
    Status TLV support (local/remote)   : enabled/supported
      LDP route watch                   : disabled
      Label/status state machine        : established, LruRru
      Last local dataplane   status rcvd: No fault
      Last BFD dataplane     status rcvd: Not sent
      Last BFD peer monitor  status rcvd: No fault
      Last local AC  circuit status rcvd: No fault
      Last local AC  circuit status sent: No fault
      Last local LDP TLV     status sent: No fault
      Last remote LDP TLV    status rcvd: No fault
      Last remote LDP ADJ    status rcvd: No fault
    MPLS VC labels: local 409, remote 309 
    AGI: type 1, len 8, 000A 0064 0000 03E8
    Local AII: type 1, len 4, 3B00 0004 (59.0.0.4)
    Remote AII: type 1, len 4, 3B00 0003 (59.0.0.3)
    Group ID: local n/a, remote n/a
    MTU: local 1500, remote 1500
    Remote interface description: 
  Sequencing: receive disabled, send disabled
  Control Word: On (configured: autosense)
  SSO Descriptor: 59.0.0.3/1000, local label: 409
  Dataplane:
    SSM segment/switch IDs: 32788/12301 (used), PWID: 4
  VC statistics:
    transit packet totals: receive 84, send 121
    transit byte totals:   receive 5592, send 10676
    transit packet drops:  receive 0, seq error 0, send 0

PE3#

In the above outputs, I have highlighted a few sections of the output. We need to ensure that those are correct like the status of the vc is UP, we need to ensure the MTU on both the sides of the vc is matching, the local and the remote vc labels and more importantly we need to ensure that the traffic is passing through. There are a few more sections in the output on ASR1k as compared to 7600 which are going to be discussed later when we will be seeing some internal details of the vc on 7600.

A new database (PW RIB) will be used to save globally unique PW endpoints (e.g. those discovered via BGP). The primary purpose of the RIB is to return a Next-Hop when a PW needs to be provisioned (added to the XC DB). Two basic functions are provided by the RIB. First, entities (i.e. BGP) can add endpoint information to the database.  The second function supports provisioning of the PW by providing an API to return the information needed to provision the PW (Pseudowire). The NH(s) are returned by this API.  Using this information, the XC DB (Xconnect Database) can be provisioned, which will initiate creation of the PW.

We shall now have a look at the outputs and understand the working of BGP-based VPLS Autodiscovery.

Output from PE1:
============
PE1#sh vfi

Legend: RT=Route-target, S=Split-horizon, Y=Yes, N=No

VFI name: vpls1, state: up, type: multipoint
  VPN ID: 1001, VPLS-ID: 100:1001    <<<<<<<<<<<<<<
  RD: 100:1001, RT: 100:1001
  Local attachment circuits:
    Vlan1001  
  Neighbors connected via pseudowires:
  Peer Address     VC ID        Discovered Router ID    S
  59.0.0.4         1001         59.0.0.4                Y
  59.0.0.3         1001         59.0.0.3                Y

VFI name: vpls2, state: up, type: multipoint
  VPN ID: 1000, VPLS-ID: 100:1000
  RD: 100:1000, RT: 100:1000
  Local attachment circuits:
    Vlan1000  
  Neighbors connected via pseudowires:
  Peer Address     VC ID        Discovered Router ID    S
  59.0.0.3         1000         59.0.0.3                Y
  59.0.0.4         1000         59.0.0.4                Y

PE1#

Output from PE3:
============
PE3#sh vfi
Legend: RT=Route-target, S=Split-horizon, Y=Yes, N=No

VFI name: vpls1, state: up, type: multipoint signaling: LDP
  VPN ID: 1001, VPLS-ID: 100:1001    <<<<<<<<<<<<<<<
  RD: 100:1001, RT: 100:1001
  Bridge-Domain 1001 attachment circuits:
  Neighbors connected via pseudowires:
  Peer Address     VC ID        Discovered Router ID    S
  59.0.0.1         1001         59.0.0.1                Y
  59.0.0.3         1001         59.0.0.3                Y

VFI name: vpls2, state: up, type: multipoint signaling: LDP
  VPN ID: 1000, VPLS-ID: 100:1000
  RD: 100:1000, RT: 100:1000
  Bridge-Domain 1000 attachment circuits:
  Neighbors connected via pseudowires:
  Peer Address     VC ID        Discovered Router ID    S
  59.0.0.3         1000         59.0.0.3                Y
  59.0.0.1         1000         59.0.0.1                Y

PE3#

From the above output, we can see that the VPN ID is the same as we configured under the l2 vfi config. the VPLS-id was not configured. This was automatically generated. We can the information specific to the VFI that we configured: vpls1 and vpls2. Also, we see the VC ID and the Discovered Router ID of the BGP peers for the particular VFI. On the ASR1k router, we also see the the signalling is done by the LDP (as mentioned above). We can also specify a different rd and rt values under the vfi and if the remote PE has the same rd and rt value configured for the VFI, the circuit will come up. Also, we discussed about the PW RIB. We can use the show xconnect rib command to view the details of the PW RIB. 

Output from PE1:
============
PE1#show xconnect rib  detail 

Local Router ID: 59.0.0.1

VPLS-ID: 100:1000, Target ID: 59.0.0.3
  Next-Hop: 59.0.0.3
  Hello-Source: 59.0.0.1
  Route-Target: 100:1000
  Incoming RD: 100:1000
  Forwarder: VFI vpls2
  Origin: BGP
  Provisioned: Yes
VPLS-ID: 100:1000, Target ID: 59.0.0.4
  Next-Hop: 59.0.0.4
  Hello-Source: 59.0.0.1
  Route-Target: 100:1000
  Incoming RD: 100:1000
  Forwarder: VFI vpls2
  Origin: BGP
  Provisioned: Yes
VPLS-ID: 100:1001, Target ID: 59.0.0.3
  Next-Hop: 59.0.0.3
  Hello-Source: 59.0.0.1
  Route-Target: 100:1001
  Incoming RD: 100:1001
  Forwarder: VFI vpls1
  Origin: BGP
  Provisioned: Yes
VPLS-ID: 100:1001, Target ID: 59.0.0.4
  Next-Hop: 59.0.0.4
  Hello-Source: 59.0.0.1
  Route-Target: 100:1001
  Incoming RD: 100:1001
  Forwarder: VFI vpls1
  Origin: BGP
  Provisioned: Yes
PE1#

Lets now have a look at few of the BGP related outputs:

Output from PE1:
============
PE1#sh ip bgp l2vpn vpls all summ
BGP router identifier 59.0.0.1, local AS number 100
BGP table version is 18, main routing table version 18
6 network entries using 888 bytes of memory
6 path entries using 360 bytes of memory
4/4 BGP path/bestpath attribute entries using 496 bytes of memory
2 BGP extended community entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 1792 total bytes of memory
BGP activity 11/5 prefixes, 11/5 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
59.0.0.3        4          100   16835   16847       18    0    0 1w3d            2
59.0.0.4        4          100   14433   14434       18    0    0 1w2d            2
PE1#

PE1#sh ip bgp l2vpn vpls all     
BGP table version is 18, local router ID is 59.0.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, x best-external
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 100:1000
*> 100:1000:59.0.0.1/96
                    0.0.0.0                            32768 ?
*>i100:1000:59.0.0.3/96
                    59.0.0.3                 0    100      0 ?
*>i100:1000:59.0.0.4/96
                    59.0.0.4                 0    100      0 ?
Route Distinguisher: 100:1001
*> 100:1001:59.0.0.1/96
                    0.0.0.0                            32768 ?
*>i100:1001:59.0.0.3/96
                    59.0.0.3                 0    100      0 ?
*>i100:1001:59.0.0.4/96
                    59.0.0.4                 0    100      0 ?
PE1#

We can similar outputs on PE2 and PE3.

So, from the outputs we can see that there are no MAC addresses getting exchanged but it the displays layer 2 NLRI for a particular VPLS instance in the VPLS address family. If we try to look at the prefix 59.0.0.1 in the l2vpn vpls table in BGP, we see the following:

Output from PE1:
============
PE1#sh ip bgp l2vpn vpls all 59.0.0.3   
BGP routing table entry for 100:1000:59.0.0.3/96, version 11
Paths: (1 available, best #1, table L2VPN-VPLS-BGP-Table)
  Not advertised to any peer
  Local
    59.0.0.3 from 59.0.0.3 (59.0.0.3)
      Origin incomplete, metric 0, localpref 100, valid, internal, best, AGI version(1577058312)
      Extended Community: RT:100:1000 L2VPN AGI:100:1000
BGP routing table entry for 100:1001:59.0.0.3/96, version 4
Paths: (1 available, best #1, table L2VPN-VPLS-BGP-Table)
  Not advertised to any peer
  Local
    59.0.0.3 from 59.0.0.3 (59.0.0.3)
      Origin incomplete, metric 0, localpref 100, valid, internal, best, AGI version(3)
      Extended Community: RT:100:1001 L2VPN AGI:100:1001
PE1#

We can look at the mac address-table on PE1 and PE2 for the vlans 1000 and 1001. We need to initiate a ping from the CE devices to the remote CE devices.

Output from PE1:
============
PE1#sh mac address-table vl 1000 all 
Legend: * - primary entry
        age - seconds since last seen
        n/a - not available

  vlan   mac address     type    learn     age              ports
------+----------------+--------+-----+----------+--------------------------
Module 4:
* 1000  000f.90ee.77c2   dynamic  Yes        120   59.0.0.4, 1000
* 1000  0024.970b.9001   dynamic  Yes          5   59.0.0.3, 1000
Active Supervisor:
  1000  000f.90ee.77c2   dynamic  Yes        120   59.0.0.4, 1000

PE1#

But just the mac table is not going to help here. Since the EVCs (service instances) are created to pass the traffic through the switch, we also need to check if the traffic is passing through the EVC.

Output on PE1:
==========
PE1#show ethernet service instance id 1000 interface gi 4/2 detail 
Service Instance ID: 1000
Associated Interface: GigabitEthernet4/2
Associated EVC: 
L2protocol forward
CE-Vlans:                                                                        
Encapsulation: dot1q 1000 vlan protocol type 0x8100
Rewrite: ingress tag pop 1 symmetric
Interface Dot1q Tunnel Ethertype: 0x8100
State: Up
EFP Statistics:
   Pkts In   Bytes In   Pkts Out  Bytes Out
       189      13556     399941   28795532
EFP Microblocks:
****************
Microblock type: Bridge-domain
Bridge-domain: 1000

Output of the EVC on the Linecard:  (PE1 - 7600)
==================================
PE1#attach 4
Entering CONSOLE for slot 4
Type "^C^C^C" to end this session

PE1-dfc4#show ethernet service instance id 1000 interface gi 4/2 detail
EFP ID: 1000
Associated Interface: GigabitEthernet4/2
State: Up
L2protocol forward
Encapsulation: dot1q 1000 vlan protocol type 0x8100
Rewrite: ingress tag pop 1 symmetric
Interface Dot1q Tunnel Ethertype: 0x8100

EFP Microblocks:
****************
Microblock type: Bridge-domain
Bridge-domain: 1000
MAC security: Disabled
PE1-dfc4#


Output on PE3:
=============
PE3#show ethernet service instance id 1000 interface gi 0/0/1 det
Service Instance ID: 1000
Service Instance Type: static
Associated Interface: GigabitEthernet0/0/1
Associated EVC: 
L2protocol drop
CE-Vlans:                                                                        
Encapsulation: dot1q 1000 vlan protocol type 0x8100
Rewrite: ingress tag pop 1 symmetric
Interface Dot1q Tunnel Ethertype: 0x8100
State: Up
EFP Statistics:
   Pkts In   Bytes In   Pkts Out  Bytes Out
    396249   26944964        174      11568
EFP Microblocks:
****************
Microblock type: Bridge-domain
Bridge-domain: 1000

Microblock type: L2Mcast
L2 Multicast GID: 2

PE3#

In the above output we see that the packets in and packets out have increased on the EVC. One important thing to remember is that sometimes there are some issues on the EVC (due to software defects) which cause the traffic to stop forwarding. So we need to ensure that the EVCs are correctly programmed on the hardware of the 7600 platform. Thats the reason we went into the Linecard having the EVC on 7600 and verified the same command and notice that the EVC is programmed on the Linecard as well. As far as ASR1k is concerned, we can check the mac-addresses and other details for the bridge-domain over which the vc is built:

Initiate a ping from CE3 to CE1 and then we can see the following output on ASR1k PE router.

Outputs from PE3:
=============
PE3#sh bridge-domain 1001
Bridge-domain 1001 (3 ports in all)
State: UP                    Mac learning: Enabled
Aging-Timer: 300 second(s)
    GigabitEthernet0/0/1 service instance 1001
    vfi vpls1 neighbor 59.0.0.1 1001
    vfi vpls1 neighbor 59.0.0.3 1001
   MAC address    Policy    Tag     Age Pseudoport
   2C54.2D78.3141 forward dynamic   270 vpls1.1020011
   000F.90EE.77C1 forward dynamic   270 GigabitEthernet0/0/1.EFP1001

Output from CE3:
============
CE3#sh int vl 1001
Vlan1001 is up, line protocol is up 
  Hardware is EtherSVI, address is 000f.90ee.77c1 (bia 000f.90ee.77c1)
  Internet address is 192.168.1.3/24
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255

. . . . 

Output from CE1:
============
CE1#sh int vl 1001
Vlan1001 is up, line protocol is up 
  Hardware is EtherSVI, address is 2c54.2d78.3141 (bia 2c54.2d78.3141)
  Internet address is 192.168.1.1/24
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
. . . .

From the above we can see on the ASR1k PE router, for the bridge-domain (BD) 1001, we see that this node has two neighbors which are PE1 and PE2. Also, we see two mac-addresses being learnt on the BD which are the same as the vlan 1001 interfaces on both CE1 and CE3 routers. 

Troubleshooting VPLS

In order to troubleshoot VPLS, lets first start by checking the Platform Independent (PI) stuff. Remember that the below steps come into picture only after the basic things have been checked which have already been showed above.

Output on PE1:
===========
PE1#show mpls l2 vc vcid 1000 ssm id

VC Peer ID      VC ID      SSM Switch SSM Seg ID
--------------- ---------- ---------- ----------
59.0.0.3        1000       16390      40970     
59.0.0.4        1000       12298      49170   

PE1#show ssm switch id 16390 
  Switch-ID 16390 State: Open   <<<<<<<<<<<<
    Segment-ID: 40970 Type: AToM[17]
      Switch-ID:                    16390
      Allocated By:                 This CPU
      Locked By:                    SIP     [1]
      Class:                        SSS
        State:                        Active
      Class:                        ADJ
        State:                        Active

    Segment-ID: 36873 Type: VFI[25]
      Switch-ID:                    16390
      Allocated By:                 This CPU
      Locked By:                    SIP     [1]
      Circuit status:               Up      [1]
      Class:                        SSS
        State:                        Active
        VFI-PW:
          Private data is not set
      Class:                        ADJ
        State:                        Active
        VFI-PW:
          VFI id: 2 bdomain 0 SVI if-num 43
          plat-idx 0xC3E88004 peer-id 59.0.0.3 SH 1


PE1#

Output from PE3: (ASR1000 router)
============
PE3#sh mpls l2 vc vcid 1000 ssm id 

VC Peer ID      VC ID      SSM Switch SSM Seg ID
--------------- ---------- ---------- ----------
59.0.0.1        1000       16399      16402     
59.0.0.3        1000       12301      32788     

PE3#show ssm switch id 16399
  Switch-ID 16399 State: Open
    Segment-ID: 16402 Type: AToM[17]
      Switch-ID:                    16399
      Allocated By:                 This CPU
      Locked By:                    SIP     [1]
    Class:                        SSS
      State:                        Active
    Class:                        ADJ
      State:                        Active
 
    Segment-ID: 24592 Type: VFI[25]
      Switch-ID:                    16399
      Allocated By:                 This CPU
      Locked By:                    SIP     [1]
    Class:                        SSS
      State:                        Active
      VFI-PW:
        Private data is not set
    Class:                        ADJ
      State:                        Active
      VFI-PW:
        VFI id: 2 bdomain 1000 SVI if-num 0
        plat-idx 0x1020018 peer-id 59.0.0.1 SH 1
  

PE3#

From the above logs, we see the non-zero values for the ssm switch id and the ssm segment id. Please ensure that these values are not zero. These values will be zero if the VC is down. Also when we look at the "show ssm switch id <>" command, please ensure that the status is "Open" and not "Down". Rest all the information is pretty much understandable.

The next step is to check the mpls l2transport bindings for the particular VC:

Output on PE1:
===========
PE1#show mpls l2transport binding 1000 
  Destination Address: 59.0.0.3,  VC ID: 1000
    Local Label:  208
        Cbit: 1,    VC Type: Ethernet,    GroupID: n/a
        MTU: 1500,   Interface Desc: n/a
        VCCV: CC Type: RA [2], TTL [3]
              CV Type: LSPV [2]
    Remote Label: 308
        Cbit: 1,    VC Type: Ethernet,    GroupID: n/a
        MTU: 1500,   Interface Desc: n/a
        VCCV: CC Type: RA [2], TTL [3]
              CV Type: LSPV [2]
  Destination Address: 59.0.0.4,  VC ID: 1000
    Local Label:  209
        Cbit: 1,    VC Type: Ethernet,    GroupID: n/a
        MTU: 1500,   Interface Desc: n/a
        VCCV: CC Type: RA [2], TTL [3]
              CV Type: LSPV [2]
    Remote Label: 408
        Cbit: 1,    VC Type: Ethernet,    GroupID: n/a
        MTU: 1500,   Interface Desc: n/a
        VCCV: CC Type: RA [2], TTL [3]
              CV Type: LSPV [2]
PE1#

Output from PE3:
============
PE3#show mpls l2transport binding 1000
  Destination Address: 59.0.0.1,VC ID: 1000
    Local Label:  408
        Cbit: 1,    VC Type: Ethernet,    GroupID: n/a
        MTU: 1500,   Interface Desc: n/a
        VCCV: CC Type: RA [2], TTL [3]
              CV Type: LSPV [2]
    Remote Label: 209
        Cbit: 1,    VC Type: Ethernet,    GroupID: n/a
        MTU: 1500,   Interface Desc: n/a
        VCCV: CC Type: RA [2], TTL [3]
              CV Type: LSPV [2]
  Destination Address: 59.0.0.3,VC ID: 1000
    Local Label:  409
        Cbit: 1,    VC Type: Ethernet,    GroupID: n/a
        MTU: 1500,   Interface Desc: n/a
        VCCV: CC Type: RA [2], TTL [3]
              CV Type: LSPV [2]
    Remote Label: 309
        Cbit: 1,    VC Type: Ethernet,    GroupID: n/a
        MTU: 1500,   Interface Desc: n/a
        VCCV: CC Type: RA [2], TTL [3]
              CV Type: LSPV [2]

PE3#

These commands will basically give the details of the local as well as remote label along with the other required details of the VC. 

Check if LTL is allocated fine. If there are any platform issues like exceeding maximum allowed VC, there will be some allocation failures seen with this command.

Output from PE1:
============
PE1#show platform ltl-mgr vlan 1000 
CWAN LTL Manager:

 Pseudo-wire Database Usage:
  CWAN LTL: Vlan:0x3E8(1000)   Count: 3  
   MPB PWs - 
      pw: slot: 4 , slotunit 284 egress_slotunit 2 
   VPLS PWs - 
      pw: slotunit 5   peer 59.0.0.4 vc 1000
      pw: slotunit 4   peer 59.0.0.3 vc 1000
PE1#

Note that this LTL-mgr only applies to the 7600 platform.

We also need to check the scalability of VPLS on the 7600 platform.

96K MAC address for RSP and 65K for SUP720. Scaling really depends on supervisor type.
Maximum number of VC that can be configured in the box is 32K.
Maximum VFI per system is 4K (Limited by max number of SVI per box)
Maximum neighbors per VFI is 110 (Including the bridge-domain configured in the box)
For some cards like SIP600, the maximum is 60 which is due to platform limitation. For ES20/ES+ it can be increased to 110.

The current configured limit can be seen as below:

Output from PE1:
============
PE1#show platform vfi max-peers 
 VFI max peers now set to 60. It will be Unchanged on next reboot

We can also change the max-peers value as below.

PE1#conf t
PE1(config)#platform vfi max-peers ?
  <110-110>   max peers value allowed 60 or 110
  <60-60>     max peers value allowed 60 or 110

If there is a packet forwarding issue in a VPLS environment, we can also perform a packet capture on PE routers like the 7600 and the ASR1k router. For packet capture on ASR1k platform, please refer to my post on the same. Below is an example of capturing a packet on the linecard on the 7600 router:

Outputs from PE1:
=============
PE1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
PE1(config)#service internal            <<<<<<<<<<<<<<<<<
PE1(config)#end
PE1#att 
*Aug 17 01:53:47.548: %SYS-5-CONFIG_I: Configured from console by console
PE1#att 4
Entering CONSOLE for slot 4
Type "^C^C^C" to end this session


PE1-dfc4>en
PE1-dfc4#show platform capture elam asic superman slot 4
PE1-dfc4#show platform capture elam trigger dbus ipv4 if ip_sa=192.168.1.3 ip_da=192.168.1.1
PE1-dfc4#show platform capture elam start
PE1-dfc4#show platform cap el stat
active ELAM info:
Slot Cpu   Asic   Inst Ver PB Elam
---- --- -------- ---- --- -- ----
4    0   ST_SMAN  0    3.2    Y    
DBUS trigger: FORMAT=IP L3_PROTOCOL=IPV4 IP_SA=192.168.1.3 IP_DA=192.168.1.1
elam capture in progress   <<<<<<<<<<<<
PE1-dfc4#

Output from CE3:
================
Ping initiated from CE3 towards CE1

CE3#ping 192.168.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/5/9 ms
CE3#

Output from PE1:
================
PE1-dfc4#show platform cap el stat
active ELAM info:
Slot Cpu   Asic   Inst Ver PB Elam
---- --- -------- ---- --- -- ----
4    0   ST_SMAN  0    3.2    Y    
DBUS trigger: FORMAT=IP L3_PROTOCOL=IPV4 IP_SA=192.168.1.3 IP_DA=192.168.1.1
elam capture completed
PE1-dfc4#

PE1-dfc4#show platform cap el stat
active ELAM info:
Slot Cpu   Asic   Inst Ver PB Elam
---- --- -------- ---- --- -- ----
4    0   ST_SMAN  0    3.2    Y    
DBUS trigger: FORMAT=IP L3_PROTOCOL=IPV4 IP_SA=192.168.1.3 IP_DA=192.168.1.1
elam capture in progress
PE1-dfc4#show platform cap el stat
active ELAM info:
Slot Cpu   Asic   Inst Ver PB Elam
---- --- -------- ---- --- -- ----
4    0   ST_SMAN  0    3.2    Y    
DBUS trigger: FORMAT=IP L3_PROTOCOL=IPV4 IP_SA=192.168.1.3 IP_DA=192.168.1.1
elam capture completed     <<<<<<<<<<
PE1-dfc4#
PE1-dfc4#show platform capture elam data
PE1-dfc4#show platform capture elam data 
DBUS data:
SEQ_NUM                          [5] = 0x13
QOS                              [3] = 0
QOS_TYPE                         [1] = 0
TYPE                             [4] = 0 [ETHERNET]
STATUS_BPDU                      [1] = 0
IPO                              [1] = 0
NO_ESTBLS                        [1] = 0
RBH                              [3] = b000
CR                               [1] = 0
TRUSTED                          [1] = 1
NOTIFY_IL                        [1] = 0
NOTIFY_NL                        [1] = 0
DISABLE_NL                       [1] = 0
DISABLE_IL                       [1] = 0
DONT_FWD                         [1] = 0
INDEX_DIRECT                     [1] = 0
DONT_LEARN                       [1] = 0
COND_LEARN                       [1] = 0
BUNDLE_BYPASS                    [1] = 0
QOS_TIC                          [1] = 0
INBAND                           [1] = 0
IGNORE_QOSO                      [1] = 0
IGNORE_QOSI                      [1] = 0
IGNORE_ACLO                      [1] = 0
IGNORE_ACLI                      [1] = 0
PORT_QOS                         [1] = 0
CACHE_CNTRL                      [2] = 0 [NORMAL]
VLAN                             [12] = 1001
SRC_FLOOD                        [1] = 0
SRC_INDEX                        [19] = 0x8EF
LEN                              [16] = 118
FORMAT                           [2] = 0 [IP]
MPLS_EXP                         [3] = 0x0
REC                              [1] = 0
NO_STATS                         [1] = 0
VPN_INDEX                        [10] = 0x0
PACKET_TYPE                      [3] = 0 [ETHERNET]
L3_PROTOCOL                      [4] = 0 [IPV4]
L3_PT                            [8] = 1 [ICMP]
MPLS_TTL                         [8] = 128
SRC_XTAG                         [4] = 0x0
DEST_XTAG                        [4] = 0x0
FF                               [1] = 0
MN                               [1] = 0
RF                               [1] = 0
SC                               [1] = 0
CARD_TYPE                        [4] = 0x0
DMAC                             = 2c54.2d78.3141
SMAC                             = 000f.90ee.77c1
IPVER                            [1] = 0 [IPV4]
IP_DF                            [1] = 0
IP_MF                            [1] = 0
IP_HDR_LEN                       [4] = 5
IP_TOS                           [8] = 0x0
IP_LEN                           [16] = 100
IP_HDR_VALID                     [1] = 1
IP_CHKSUM_VALID                  [1] = 1
IP_L4HDR_VALID                   [1] = 1
IP_OFFSET                        [13] = 0
IP_TTL                           [8] = 255
IP_CHKSUM                        [16] = 0x304A
IP_SA                            = 192.168.1.3
IP_DA                            = 192.168.1.1
ICMP_TYPE                        [8] = 0x8
ICMP_CODE                        [8] = 0x0
ICMP_DATA [104]
0000:  16 55 00 0B 00 00 00 00 00 00 65 AB CD            ".U........e.."
CRC                              [16] = 0xEB3B
RBUS data:
SEQ_NUM                          [5] = 0x13
CCC                              [3] = b000 [NO_REWRITE]
CAP1                             [1] = 0
CAP2                             [1] = 0
QOS                              [3] = 0
EGRESS                           [1] = 0
DT                               [1] = 1 [GENERIC]
TL                               [1] = 0 [B32]
FLOOD                            [1] = 1
DEST_INDEX                       [19] = 0x3E9   <<<<<<<<<<<
VLAN                             [12] = 1001
RBH                              [3] = b010
RDT                              [1] = 0
GENERIC                          [1] = 0
EXTRA_CICLE                      [1] = 0
FABRIC_PRIO                      [1] = 0
L2                               [1] = 0
FCS1                             [8] = 0x1
DELTA_LEN                        [8] = 0
REWRITE_INFO
 i0  - no rewrite.
FCS2                             [8] = 0x0
PE1-dfc4#

In order to perform the elam capture, first you need to enable "service internal" command. Now, lets first understand what ELAM is?

The Embedded Logic Analyzer Module (ELAM) is an hardware module, embedded on the major Constellation ASICs to give improved diagnostic access to internal and external signals. ASIC signals are internally captured and routed into the ELAM module. The ELAM has a powerful and flexible triggering capability that determines which data is captured. The module can trigger based on data inputs, external trigger inputs, and can also drive external trigger outputs. Superman has 3 ELAM modules, capable to capture data coming from the Constellation Data Bus interface (DBUS), partial results coming from the L3 Forwarding Engine (Tycho), and final results transmitted on the Constellation Result Bus (RBUS). We shall be further discussing about the ASIC and other stuff in our Networks blog which discusses about the platform related stuff. 

Now, in the above output, we see that the Source and Desitnation IP is the same as we mentioned. The more important stuff is the DEST_INDEX value in the RBus output. This value points out to the internal hardware programming which says where this packet is supposed to be sent. We can see the details of the DEST_INDEX value using the test mcast command from the Switch Processor (SP) of the 7600 router.

Output from PE1:
============
PE1#remote command switch test mcast ltl-info index 3E9

index 0x3E9 contain ports 4/T1,T2
PE1#

The above output points to the Titan ASIC of the slot 4. Sometimes this ltl value is mis-programmed in hardware which leads to packet forwarding issue.

Hope you enjoyed this post. Please feel free to reach out to me for any questions, suggestion or feedback.

Cheers...!!!

Comments (4) -

  • Pablo

    8/8/2013 7:28:52 AM |

    Great article, I really enjoyed the read. Do you work in TAC? The only time I have heard of anyone using these features were TAC Engineers.

    Thank you for your contributions, they have helped me out while studying for the CCIE SP lab.  

  • Genie

    8/11/2013 9:55:52 AM |

    Hello Pablo
    Thanks for such nice comments. I will ensure that I maintain the quality of the posts. We provide solutions to different SP's, help our customers design and implement those solutions in their network. Also, we have our own research team who does a lot of RnD on Cisco devices and get handy with them.
    keep in touch and feel free to write to me in case you need any help.

    Always Happy to Help.

  • Guilherme

    3/31/2014 9:32:54 PM |

    hello Genie

    congratulations on your article, many took my questionswanted to know if you could help me in a scenario between a 7606 (RSP720-3C-GE and 76-ES + XC-20G3C) with c7600rsp72043-adventerprisek9-mz.153-1.S1 IOS , and ME3600.

      the command in 7606:

    switch # show ssm id 8909

    shows the following: Switch-ID 8909 State: Down

    and the command at ME3600:

    switch # show ssm id 28678
       Switch-ID 28678 State: Open

    the rest seems to be all right, will be who can help me with that?

  • Genie

    4/1/2014 6:42:34 AM |

    Hello Guilherme
    This blog is just for discussing technology, we have a separate support forum where in we look at the problems that you are facing.
    Could you please post your questions at http://forums.codergenie.com
    The registration is completely free.

Comments are closed