Genie's Tech Blog

Where knowledge has no dimensions

Multicast VPN (mVPN)

Hello Friends,

Today, I will be discussing about Multicast VPN (MVPN) in which I will be explaining how it works with an example and help you troubleshoot mVPN issues on 7600 platform. As a pre-requisite, I assume everyone understands basic multicast concepts and different PIM modes specially PIM SP (Sparce Mode) and also Multicast VPN concept. First of all we, its impotant to understand that Multicast forwarding is opposite of Unicast forwarding. Unicast is concerned about where the packet is going where as Multicast is concerned about where the packet came from. Multicast uses Reverse Path Forwarding (RPF) which checks if the arriving packet is on reverse path back to the source. If successful, packets are fowarded else they are dropped. RPF uses the underlying Multicast protocol like PIM / DMVPN to verify the reverse path forwarding. As we are aware that PIM is having different modes, we will be suing PIM Sparce Mode in the core with Auto-RP as the mechanism for figuring out the randezvous point (SP) with the core and will be using static RP as the RP mechanism for PE-CE.

Multicast VPN implementation:

The CE devices maintain PIM adjacency with the PE router only. The provider core network does not need to hold (S, G) for individual customers. Since customers multicast groups can coverlap, they will be segregated under different customer vrf's. All the Multicast VPN configuration is done on the PE routers only. The mVPN customers are unaffected by SP core i.e. the core is completely independent of the customer multicast operation. SP core just requires native multicast or MPLS (for mLDP) to carry the customer Multicast traffic. If we run native multicast, we just need PIM SM or SSM (Source specific Multicast).

PE routers build a default Multicast Distribution Tree (MDT) in the global table for each of its mVRF (Multicast enabled VRF's) using standard PIM-SM or SSM procedures. All the PE routers participating in the same mVPN join the same Default MDT. Please note that every mVRF must have a Default-MDT. MDT group addresses are defined by the provider which are unrelated to the groups used by the customer. Default MDT is used as a permanent channel both PIM control messages and low bandwidth streams. Access to the Default MDT from the mVRF is via a Multicast Tunnel Interface (MTI) which appears as a “TunnelX” interface in the mVRF. A PE is always a root (source) of the MDT. A PE is also a leaf (receiver) to the MDT rooted on remote PEs. All (S, G), (*, G) entries in the customer VPN get mapped to a single Service Provider MDT-group. Optionally a Data MDT can created from sending PE when a high bandwidth source appears in the customer network. Data MDTs will be created for customer (S, G) states only.Trees are optimised for the source and active receivers. MTI is  configurable and takes its properties from interface used for BGP peering (e.g. Loopback 0). Its imoptant to know that PIM-SM/DM always enabled on the MTI and no unicast runs over MTI. At present GRE is the only method available. Once a packet is forwarded to the MTI is passes into the global multicast of the SP MTI automatically created when Default-MDT configured. Lets now understand how the packet forwarding happens on the CE as well as the Provider Core network

 

Forwarding C-packets (from CE):

1. A C-Packet arrives on an VRF configured PE interface, mVRF is implicitly identified. Normal RPF check on C-source
2. The C-packet is replicated out interfaces in the olist. At this point this would be PE interfaces in the same VRF
3. If olist contains an MTI, then C-packet encapsulated into a P-packet.  If “y” flag is set on entry, destination used is Data-MDT group otherwise Default-MDT group. Source is PE BGP peer address. Destination is the MDT Group address
4. The P-packet is forwarded through P-network as per normal multicast.

 

Forwarding P-packets (from P-network)

1. P-packet arrives from global interface. Global (S, G) or (*, G) entry for the MDT-group referenced. Normal RPF check on P-source (PE peer)
2. The P-packet is replicated out interfaces in the olist. At this point this would be P/PE interfaces in the global mrouting table
3. If Z flag is set, then P-packet is decapsulated to reveal the C-packet. The target mVRF and incoming interface (MTI) is derived from MDT-group
4. RPF check of C-packet in mVRF done, C-packet replicated out olist in mVRF

I think the above stated has given a fair piece of understanding on how mVPN works. Lets now see with the help of an example in-fact a live Multicast network which we have simulated using IXIA and 7600 routers as the SP network. Lets consider the following topology:

 

Between the 7600_PE1 and IXIA, We are running OSPF and PIM. OSPF is the PE-CE protocol, Core is using ISIS as the IGP. We are also using MPLS TE in the core. Lets now have a look at the configuration of the PE routers.

Config on PE1:
==========
ip vrf ABC
 rd 1:1   
 mdt default 224.1.1.1
 mdt data 224.2.2.2 0.0.0.0 threshold 1
 route-target export 1:1
 route-target import 1:1
!
ip multicast-routing 
ip multicast-routing vrf ABC 
!
mpls label protocol ldp
no mpls ip propagate-ttl forwarded
mpls ldp label
 allocate global host-routes
mpls traffic-eng tunnels
mpls traffic-eng logging lsp path-errors
mpls traffic-eng logging lsp reservation-errors
mpls traffic-eng logging lsp setups
mpls traffic-eng logging lsp teardowns
mpls traffic-eng logging tunnel path change
mpls traffic-eng auto-tunnel backup
mpls traffic-eng auto-tunnel backup tunnel-num min 5000 max 6000
mpls traffic-eng auto-tunnel mesh
mpls traffic-eng auto-tunnel mesh tunnel-num min 1000 max 2000
mpls traffic-eng reoptimize timers frequency 1800
!
interface Auto-Template1
 ip unnumbered Loopback0
 mpls ip
 mpls mtu 17868
 tunnel mode mpls traffic-eng
 tunnel destination access-list 25
 tunnel mpls traffic-eng autoroute announce
 tunnel mpls traffic-eng priority 4 4
 tunnel mpls traffic-eng path-option 10 dynamic
 tunnel mpls traffic-eng fast-reroute
!
interface Loopback0
 ip address 10.1.0.1 255.255.255.255
 ip router isis 
 ip pim sparse-mode
 isis circuit-type level-2-only
!
interface GigabitEthernet1/1
 mtu 4470
 ip address 12.12.12.1 255.255.255.252
 ip router isis 
 ip pim sparse-mode
 mpls traffic-eng tunnels
 bfd interval 250 min_rx 250 multiplier 3
 isis circuit-type level-2-only
 isis network point-to-point 
 isis metric 15 level-2
 isis hello-interval 4
 isis csnp-interval 10
 hold-queue 4096 in
 hold-queue 4096 out
 ip rsvp bandwidth 1000000
 ip rsvp signalling hello bfd
!         
interface GigabitEthernet1/2
 ip vrf forwarding ABC
 ip address 10.11.12.1 255.255.255.0
 ip pim sparse-mode
 load-interval 30
!         
interface GigabitEthernet1/3
 mtu 4470 
 ip address 13.13.13.1 255.255.255.252
 ip router isis 
 ip pim sparse-mode
 mpls traffic-eng tunnels
 bfd interval 250 min_rx 250 multiplier 3
 isis circuit-type level-2-only
 isis network point-to-point 
 isis metric 15 level-2
 isis hello-interval 4
 hold-queue 4096 in
 hold-queue 4096 out
 ip rsvp bandwidth 1000000
 ip rsvp signalling hello bfd
!    
router ospf 100 vrf ABC
 redistribute bgp 100 subnets
 network 10.11.12.1 0.0.0.0 area 0
!
router isis
 net 49.0100.1111.1111.00
 is-type level-2-only
 metric-style wide
 set-overload-bit on-startup wait-for-bgp
 spf-interval 10 100 1000
 prc-interval 10 100 1000
 no hello padding
 log-adjacency-changes
 distance 80 ip
 mpls traffic-eng router-id Loopback0
 mpls traffic-eng level-2
 mpls traffic-eng multicast-intact
!
router bgp 100
 bgp router-id 10.1.0.1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 10.4.0.1 remote-as 100
 neighbor 10.4.0.1 update-source Loopback0
 !
 address-family ipv4
  neighbor 10.4.0.1 activate
 exit-address-family
 !        
 address-family vpnv4
  neighbor 10.4.0.1 activate
  neighbor 10.4.0.1 send-community both
  neighbor 10.4.0.1 next-hop-self
 exit-address-family
 !
 address-family ipv4 mdt
  neighbor 10.4.0.1 activate
  neighbor 10.4.0.1 send-community extended
 exit-address-family
 !
 address-family ipv4 vrf ABC
  redistribute ospf 100 match internal external 1 external 2
 exit-address-family
!
ip pim vrf ABC rp-address 10.11.12.1
ip rsvp signalling refresh reduction
ip rsvp signalling hello
ip rsvp signalling hello bfd
ip route 0.0.0.0 0.0.0.0 10.122.163.1
!
ip access-list extended ssh
 permit ip any any
!
access-list 25 permit 10.0.0.0 0.255.255.255
!
mpls ldp router-id Loopback0 force


Config on PE2:
==========
ip vrf ABC
 rd 1:1
 mdt default 224.1.1.1
 mdt data 224.2.2.2 0.0.0.0 threshold 1
 route-target export 1:1
 route-target import 1:1
!
ip multicast-routing 
ip multicast-routing vrf ABC 
!
!
!
!
no mpls ip propagate-ttl forwarded
mpls ldp label
 allocate global host-routes
mpls traffic-eng tunnels
mpls traffic-eng logging lsp path-errors
mpls traffic-eng logging lsp reservation-errors
mpls traffic-eng logging lsp setups
mpls traffic-eng logging lsp teardowns
mpls traffic-eng logging tunnel path change
mpls traffic-eng auto-tunnel backup
mpls traffic-eng auto-tunnel backup tunnel-num min 5000 max 6000
mpls traffic-eng auto-tunnel mesh
mpls traffic-eng auto-tunnel mesh tunnel-num min 1000 max 2000
mpls traffic-eng reoptimize timers frequency 1800
mpls label protocol ldp
clns routing
!
interface Auto-Template1
 ip unnumbered Loopback0
 mpls ip  
 tunnel mode mpls traffic-eng
 tunnel destination access-list 25
 tunnel mpls traffic-eng autoroute announce
 tunnel mpls traffic-eng priority 4 4
 tunnel mpls traffic-eng path-option 10 dynamic
 tunnel mpls traffic-eng fast-reroute
!         
interface Loopback0
 ip address 10.4.0.1 255.255.255.255
 ip router isis 
 ip pim sparse-mode
 isis circuit-type level-2-only
!         
interface GigabitEthernet2/1
 mtu 4470 
 ip address 34.34.34.2 255.255.255.252
 ip router isis 
 ip pim sparse-mode
 mpls traffic-eng tunnels
 bfd interval 250 min_rx 250 multiplier 3
 isis circuit-type level-2-only
 isis network point-to-point 
 isis metric 15 level-2
 isis hello-interval 4
 hold-queue 4096 in
 hold-queue 4096 out
 ip rsvp bandwidth 1000000
 ip rsvp signalling hello bfd
!         
interface GigabitEthernet2/2
 mtu 4470 
 ip address 42.42.42.2 255.255.255.252
 ip router isis 
 ip pim sparse-mode
 mpls traffic-eng tunnels
 bfd interval 250 min_rx 250 multiplier 3
 isis circuit-type level-2-only
 isis network point-to-point 
 isis metric 15 level-2
 isis hello-interval 4
 hold-queue 4096 in
 hold-queue 4096 out
 ip rsvp bandwidth 1000000
 ip rsvp signalling hello bfd
! 
interface GigabitEthernet6/1
 ip vrf forwarding ABC
 ip address 20.21.22.1 255.255.255.0
 ip pim sparse-mode
!   
router ospf 100 vrf ABC
 redistribute bgp 100 subnets
 network 20.21.22.1 0.0.0.0 area 0
!
router isis
 net 49.0100.4444.4444.00
 is-type level-2-only
 metric-style wide
 set-overload-bit on-startup wait-for-bgp
 spf-interval 10 100 1000
 prc-interval 10 100 1000
 no hello padding
 log-adjacency-changes
 distance 80 ip
 mpls traffic-eng router-id Loopback0
 mpls traffic-eng level-2
 mpls traffic-eng multicast-intact
!
router bgp 100
 bgp router-id 10.4.0.1
 no bgp default ipv4-unicast
 bgp log-neighbor-changes
 neighbor 10.1.0.1 remote-as 100
 neighbor 10.1.0.1 update-source Loopback0
 !        
 address-family ipv4
  no auto-summary
 exit-address-family
 !        
 address-family vpnv4
  neighbor 10.1.0.1 activate
  neighbor 10.1.0.1 send-community both
  neighbor 10.1.0.1 next-hop-self
 exit-address-family
 !        
 address-family ipv4 mdt
  neighbor 10.1.0.1 activate
  neighbor 10.1.0.1 send-community extended
 exit-address-family
 !        
 address-family ipv4 vrf ABC
  redistribute ospf 100 vrf ABC match internal external 1 external 2
 exit-address-family
!     
ip pim autorp listener
ip pim send-rp-announce Loopback0 scope 16
ip pim send-rp-discovery Loopback0 scope 16
ip pim vrf ABC rp-address 10.11.12.1
ip rsvp signalling refresh reduction
ip rsvp signalling hello
ip rsvp signalling hello bfd
!         
access-list 25 permit 10.0.0.0 0.255.255.255
!

I have not shared the configuration for the P routers as its only running basic IGP, native Multicast and haven configured with MPLS TE Auto-Tunnels. From the above configuration, we noticed that for the CE VRF ABC, the rp-address has been configured as 10.11.12.1 which is the CE facing IP on 7600_PE1 router. In the core, we are using Auto-RP and loopback 0 of 7600_PE2 i.e. 10.4.0.1 is acting the RP. The default MDT has been defined with the multicast address 224.1.1.1 where as the MDT Data group has been defined as 224.2.2.2. The Mcast_CE device has been configured with OSPF and loopback 0 ip which is 1.1.1.1/32 and is also having a static join to the Multicast stream 239.10.11.12 which is being sent from IXIA.

Config on Mcast_CE:
==============
ip multicast-routing
!
interface Loopback0
 ip address 1.1.1.1 255.255.255.255
 ip pim sparse-mode
 ip igmp join-group 239.10.11.12
!
interface FastEthernet0/1
 ip address 20.21.22.2 255.255.255.0
 ip pim sparse-mode
 load-interval 30
 duplex auto
 speed auto
!
router ospf 100
 network 1.1.1.1 0.0.0.0 area 1
 network 20.21.22.2 0.0.0.0 area 0
!

Also, please note that the address-family mdt under BGP gets automatically configured based on the MDT configuration under the vrf and remote PE vrf MDT configuration. Lets now have a look at some of the outputs which will show the routing table over the MPLS VPN setup and flow of the multicast stream over the VPN.

Output from Mcast_CE:
================
Mcast_CE#sh ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is not set

      1.0.0.0/32 is subnetted, 1 subnets
C        1.1.1.1 is directly connected, Loopback0
      10.0.0.0/24 is subnetted, 1 subnets
O IA     10.11.12.0 [110/2] via 20.21.22.1, 02:38:59, FastEthernet0/1
      20.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        20.21.22.0/24 is directly connected, FastEthernet0/1
L        20.21.22.2/32 is directly connected, FastEthernet0/1
Mcast_CE#

Mcast_CE#sh ip mroute 239.10.11.12
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report, 
       Z - Multicast Tunnel, z - MDT-data group sender, 
       Y - Joined MDT-data group, y - Sending to MDT-data group, 
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
 Timers: Uptime/Expires
 Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 239.10.11.12), 03:22:16/stopped, RP 10.11.12.1, flags: SJCL
  Incoming interface: FastEthernet0/1, RPF nbr 20.21.22.1
  Outgoing interface list:
    Loopback0, Forward/Sparse, 03:22:15/00:02:46

(10.11.12.2, 239.10.11.12), 02:45:24/00:02:05, flags: LJT
  Incoming interface: FastEthernet0/1, RPF nbr 20.21.22.1
  Outgoing interface list:
    Loopback0, Forward/Sparse, 02:45:24/00:02:46

Mcast_CE#
Mcast_CE#sh ip mfib active 
Active Multicast Sources - sending >= 4 kbps
Default
Group: 239.10.11.12
  Source: 10.11.12.2,
   SW Rate: 20 pps/7 kbps(1sec), 18 kbps(last 9970 sec)

Mcast_CE#


Output on 7600_PE1:
===============
7600_PE1#sh ip mroute vrf ABC 239.10.11.12
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report, 
       Z - Multicast Tunnel, z - MDT-data group sender, 
       Y - Joined MDT-data group, y - Sending to MDT-data group, 
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
 Timers: Uptime/Expires
 Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 239.10.11.12), 03:56:11/00:02:38, RP 10.11.12.1, flags: S
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    Tunnel3, Forward/Sparse, 02:41:25/00:02:38

(10.11.12.2, 239.10.11.12), 03:56:11/00:02:53, flags: Ty
  Incoming interface: GigabitEthernet1/2, RPF nbr 10.11.12.2
  Outgoing interface list:
    Tunnel3, Forward/Sparse, 02:41:25/00:03:08

7600_PE1#
7600_PE1#sh ip mfib vrf ABC 239.10.11.12 active 
Active Multicast Sources - sending >= 4 kbps
VRF ABC
Group: 239.10.11.12
  Source: 10.11.12.2,
   SW Rate: 0 pps/0 kbps(1sec), 0 kbps(last 14216 sec)
   HW Rate: 20 pps/7 kbps(1sec)

7600_PE1#
7600_PE1#sh ip pim mdt 
  * implies mdt is the default MDT
  MDT Group/Num   Interface   Source                   VRF
* 224.1.1.1       Tunnel3     Loopback0                ABC
7600_PE1#
7600_PE1#sh ip mroute active 
Use "show ip mfib active" to get better response time for a large number of mroutes.

Active IP Multicast Sources - sending >= 4 kbps

Group: 224.2.2.2, (?)
   Source: 10.1.0.1 (?)
     Rate: 20 pps/11 kbps(1sec), 11 kbps(last 0 secs), 2432 kbps(life avg)
7600_PE1#
7600_PE1#sh ip mroute 224.1.1.1
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report, 
       Z - Multicast Tunnel, z - MDT-data group sender, 
       Y - Joined MDT-data group, y - Sending to MDT-data group, 
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
 Timers: Uptime/Expires
 Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 224.1.1.1), 03:57:39/stopped, RP 10.4.0.1, flags: SJCFZ
  Incoming interface: GigabitEthernet1/1, RPF nbr 12.12.12.2
  Outgoing interface list:
    MVRF ABC, Forward/Sparse, 03:57:39/00:02:00

(10.4.0.1, 224.1.1.1), 03:33:34/00:01:23, flags: JTZ
  Incoming interface: GigabitEthernet1/1, RPF nbr 12.12.12.2
  Outgoing interface list:
    MVRF ABC, Forward/Sparse, 03:33:34/00:02:25

(10.1.0.1, 224.1.1.1), 03:57:39/00:02:44, flags: FT
  Incoming interface: Loopback0, RPF nbr 0.0.0.0
  Outgoing interface list:
    GigabitEthernet1/1, Forward/Sparse, 03:28:48/00:03:15

7600_PE1#
7600_PE1#sh ip mroute 224.2.2.2
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report, 
       Z - Multicast Tunnel, z - MDT-data group sender, 
       Y - Joined MDT-data group, y - Sending to MDT-data group, 
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
 Timers: Uptime/Expires
 Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 224.2.2.2), 02:47:04/stopped, RP 10.4.0.1, flags: SPFz
  Incoming interface: GigabitEthernet1/1, RPF nbr 12.12.12.2
  Outgoing interface list: Null

(10.1.0.1, 224.2.2.2), 02:47:01/00:02:28, flags: FTz
  Incoming interface: Loopback0, RPF nbr 0.0.0.0
  Outgoing interface list:
    GigabitEthernet1/1, Forward/Sparse, 02:47:01/00:03:03

7600_PE1#

Outputs on 7600_PE2:
===============
7600_PE2#sh ip mroute vrf ABC 239.10.11.12 
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report, 
       Z - Multicast Tunnel, z - MDT-data group sender, 
       Y - Joined MDT-data group, y - Sending to MDT-data group, 
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
 Timers: Uptime/Expires
 Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 239.10.11.12), 03:28:52/00:02:35, RP 10.11.12.1, flags: S
  Incoming interface: Tunnel3, RPF nbr 10.1.0.1
  Outgoing interface list:
    GigabitEthernet6/1, Forward/Sparse, 02:44:21/00:02:35

(10.11.12.2, 239.10.11.12), 02:49:41/00:01:38, flags: TY
  Incoming interface: Tunnel3, RPF nbr 10.1.0.1, MDT:224.2.2.2/00:02:12
  Outgoing interface list:
    GigabitEthernet6/1, Forward/Sparse, 02:44:22/00:02:38

7600_PE2#
7600_PE2#sh ip mfib vrf ABC ac
Active Multicast Sources - sending >= 4 kbps
VRF ABC
Group: 239.10.11.12
  Source: 10.11.12.2,
   SW Rate: 0 pps/0 kbps(1sec), 0 kbps(last 10220 sec)
   HW Rate: 20 pps/7 kbps(1sec)

7600_PE2#

Please note that in the above outputs, instead of using show ip mfib <group> active, we can also use show ip mroute <group> active but this command is getting deprecated and in the later versions of IOS, we will have to use show ip mfib active to check the rate of multicast traffic no the router. If we look at the outputs on 7600_PE1, we can see that the incoming interface for the CE Multicast group 239.10.11.12 is Gig 1/2 where as the outgoing interface is the Tunnel interface (Tunnel 3) which is the MDT tunnel. Also, since the rate of traffic is high than the threshold value configured under the vrf, the multicast traffic is flowing across the SP core on the multlcast address 224.2.2.2. The Default MDT will only be used for PIM hellos or any other control messages. You can check the information about the mdt using the show ip pim mdt command. In order for a successful flow of a multicast stream, there should always be a OIL (Outgoing Interface List).

Since we have configure mVPN on 7600 so there will be a lot of information which we need to check on the hardware front incase we run into any issues. First of all, we need to check what is the VPN id of the VRF and what is the internal vlan that is mapped to that VPN id. This is also important to know if you are troublshooting the issue from the encapsulation side or decapsulation side.

Output from 7600_PE1:
================
7600_PE1#sh vrf detail ABC
VRF ABC (VRF Id = 1); default RD 1:1; default VPNID <not set>   <<<<<<<
  Interfaces:
    Gi1/2                    Tu0                      Tu1                     
    Tu3                     
Address family ipv4 (Table ID = 1 (0x1)):
  Export VPN route-target communities
    RT:1:1                  
  Import VPN route-target communities
    RT:1:1                  
  No import route-map
  No export route-map
  VRF label distribution protocol: not configured
  VRF label allocation mode: per-prefix
    vrf-conn-aggr for connected and BGP aggregates (Label 16)
Address family ipv6 not active

7600_PE1#sh vlan internal usage | in VPN 1
1020 Multicast VPN 1 QOS Vlan
1031 VPN 1 Encap Vlan
1032 VPN 1 Decap Vlan

In the above output, vlan 1031 is being used for encapsulation and vlan 1032 is for decapsulation. Its also important to know on which Linecard or RP/SP, the hardware programming is happening for the mVPN.

Output from 7600_PE1:
=================
7600_PE1#sh mfib linecard 

IPv4 MFIB
 Slot     Linecard status    Broker status
 5/0      sync               enabled
 1/0      inactive           inactive

IPv6 MFIB
 Slot     Linecard status    Broker status

IPv4:ABC, 11 entries, 17 ioitems
Slot      Table state
5/0       Sync             

IPv4:Default, 15 entries, 35 ioitems
Slot      Table state
5/0       Sync

In the above output, we can notice that the slot 5 is synced and Broker status is enabled. Since slot 5 is the Supervisor card, so the programming will be happening there. Since the slot 1 is a CFC (Centralized Forwarding Card) card and not a DFC (Distributed Forwarding Card), all the hardware programming for the traffic on slot 1 will happen on the SP and not on the card itself. If you are troubleshooting PIM issues over the MDT, its important to know what are the neighbors over the vrf on the PE router.

Output on 7600_PE1:
===============
7600_PE1#sh ip pim vrf ABC neighbor 
PIM Neighbor Table
Mode: B - Bidir Capable, DR - Designated Router, N - Default DR Priority,
      P - Proxy Capable, S - State Refresh Capable, G - GenID Capable
Neighbor          Interface                Uptime/Expires    Ver   DR
Address                                                            Prio/Mode
10.11.12.2        GigabitEthernet1/2       04:24:36/00:01:44 v2    0 / G
10.4.0.1          Tunnel3                  05:09:33/00:01:33 v2    1 / DR S P G
7600_PE1#

You should always be able to see the PIM neighborship over the MDT tunnel. PIM neighbourship is established by exchanging PIM hellos through the MDT tunnel. The PIM hellos are generated at Platform Independent(PI) process and all the encap for the PIM packet are done at the process level itself. So usually the PIM neighbourship problem is due to some misprogramming at HW on the side where PIM neighbouship is not seen because the decap is done at HW level and then punted the packet to RP. So further debugging steps are all specific to decap side.

Note: If you need confirmation, you could enable "debug ip pim vrf <\vrf name> hello" on both encap and decap sides and check. Lets now check few outputs on the decapsulation side which is the 7600_PE2 router. First of all we need to check whether the code MDT tree is completely built. If core MDT not built properly, then debug the IP mcast core and mostly it is not HW issue. We can check this using the "show ip mroute <default_mdt_grp> verbose" command.

Output on 7600_PE2:
===============
7600_PE2#sh ip mroute 224.1.1.1 verbose 
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report, 
       Z - Multicast Tunnel, z - MDT-data group sender, 
       Y - Joined MDT-data group, y - Sending to MDT-data group, 
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
 Timers: Uptime/Expires
 Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 224.1.1.1), 05:14:37/00:03:25, RP 10.4.0.1, flags: SJCZ
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    MVRF ABC, Forward/Sparse, 05:14:37/00:00:12
    GigabitEthernet2/1, Forward/Sparse, 05:14:17/00:03:25

(10.1.0.1, 224.1.1.1), 05:14:29/00:02:48, flags: TZ
  Incoming interface: GigabitEthernet2/1, RPF nbr 34.34.34.1
  Outgoing interface list:
    MVRF ABC, Forward/Sparse, 05:14:29/00:00:12
          
(10.4.0.1, 224.1.1.1), 05:14:39/00:03:14, flags: T
  Incoming interface: Loopback0, RPF nbr 0.0.0.0
  Outgoing interface list:
    GigabitEthernet2/1, Forward/Sparse, 05:14:19/00:03:23

7600_PE2#

We shall now check the traffic rates and the hardware programming of the default as well as data MDT.

Output on 7600_PE2:
===============
7600_PE2#sh ip mfib 10.1.0.1 224.1.1.1 verbose 
Entry Flags:    C - Directly Connected, S - Signal, IA - Inherit A flag,
                ET - Data Rate Exceeds Threshold, K - Keepalive
                DDE - Data Driven Event, HW - Hardware Installed
I/O Item Flags: IC - Internal Copy, NP - Not platform switched,
                NS - Negate Signalling, SP - Signal Present,
                A - Accept, F - Forward, RA - MRIB Accept, RF - MRIB Forward,
                MA - MFIB Accept
Platform per slot HW-Forwarding Counts: Pkt Count/Byte Count
Platform Entry flags: HF - Hardware Forwarding, NP - Not platform switched,
                      PF - Partial Hardware Forwarding
Platform Interface flags: HW - Hardware Switched, NP - Not platform switched
Forwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kbits per second
Other counts:      Total/RPF failed/Other drops
I/O Item Counts:   FS Pkt Count/PS Pkt Count
Default
 (10.1.0.1,224.1.1.1) Flags: K HW DDE
   Platform Flags:  HW
   Slot 5: HW Forwarding: 6398310/447883688, Platform Flags:  HF MT  <<<<
   SW Forwarding: 1/0/56/0, Other: 0/0/0
   HW Forwarding:   6398327/0/70/0, Other: 0/0/0   <<<<<<<
   GigabitEthernet2/1 Flags: RA A MA
     Platform Flags: 
   Tunnel3, MDT Decap Flags: RF F NS   <<<<<<<<
     Platform Flags:  HW
     CEF: OCE (tunnel decap)
     Pkts: 0/1

7600_PE2#sh ip mfib 10.1.0.1 224.2.2.2 verb    
Entry Flags:    C - Directly Connected, S - Signal, IA - Inherit A flag,
                ET - Data Rate Exceeds Threshold, K - Keepalive
                DDE - Data Driven Event, HW - Hardware Installed
I/O Item Flags: IC - Internal Copy, NP - Not platform switched,
                NS - Negate Signalling, SP - Signal Present,
                A - Accept, F - Forward, RA - MRIB Accept, RF - MRIB Forward,
                MA - MFIB Accept
Platform per slot HW-Forwarding Counts: Pkt Count/Byte Count
Platform Entry flags: HF - Hardware Forwarding, NP - Not platform switched,
                      PF - Partial Hardware Forwarding
Platform Interface flags: HW - Hardware Switched, NP - Not platform switched
Forwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kbits per second
Other counts:      Total/RPF failed/Other drops
I/O Item Counts:   FS Pkt Count/PS Pkt Count
Default
 (10.1.0.1,224.2.2.2) Flags: K HW DDE
   Platform Flags:  HW
   Slot 5: HW Forwarding: 41059137/2874139590, Platform Flags:  HF MT <<<<
   SW Forwarding: 113/0/81/0, Other: 0/0/0
   HW Forwarding:   41059137/20/70/10, Other: 0/0/0  <<<<<<
   GigabitEthernet2/1 Flags: RA A MA
     Platform Flags: 
   Tunnel3, MDT Decap Flags: RF F NS   <<<<<<<
     Platform Flags:  HW
     CEF: OCE (tunnel decap)
     Pkts: 0/113

In the above output, I have highlighted 3 sections, One is the hardware forwarding counters, other is the more detailed information on the hardware fowarding counters which includes pps as well as kbps information of packets. Also, we need to ensure that the MDT tunnel is seen as the OIF. Now attach to the card which does HW switching for the packets coming in from the core. If the card is a CFC, then attach it to the active SP (Switching Processor) of the Sup card.

Output from 7600_PE2:
================
If SRE or above: "show platform software multicast ip cmfib <remote-PE-IP> <mdt-default> verbose". If SRD or below: "show mls cef ip multicast verbose"

7600_PE2-sp#show platform software multicast ip cmfib cmfib 10.1.0.1 224.2.2.2 verbose 
Multicast CEF Entries for VPN#0
(10.1.0.1, 224.2.2.2) 
        IOSVPN:0    (1) PI:1 (1) CR:0 (1) Recirc:0 (1)
        Vlan:1019 AdjPtr:212994 FibRpfNf:1 FibRpfDf:1 FibAddr:0x3006A
        rwvlans:1019 rwindex:0x7FFA adjmac:001c.b0b4.28c0 rdt:1 E:0 CAP1:0
        fmt:Mcast l3rwvld:1 DM:0 mtu:9238 rwtype:L3 met2:0x0 met3:0x8
        packets:0000041075453 bytes:000000003614639864
        Starting Offset: 0x0008
           V E C:1029      <<<<<<<<<<<<<<<   

        Annotation-data: [0x5D457F64]
        A-vlan: 1019 NS-vlan: 0 RP-rpf-vlan: 0
        Anntn flags: [0x100010]  H MT
        MTU: 9220 Retry-count: 0
        Sec-entries count: 0
        Met-handle: 0x44A0AA94 New-Met-handle: 0x0
        Met2-handle: NULL

        HAL L3-data : [0x49E18484]
        Flags: 0x4 FIB-index: 0x86D ADJ-index: 0x34002 NF-addr: 0xFFFFFFFF
        ML3 entry type: 0x0 [(S,G) shortcut]
        Flags: 0xA1000000 Vpn: 0 Rpf: 1019 Rw_index: 0x7FFA
        Adj_mtu: 9234 Met2: 0x0 Met3: 0x8
        V6-data: NULL

        ---LSM entries---

Found 1 entries. 1 are mfd entries
7600_PE2-sp#


7600_PE2#sh vlan int usage | in VPN 1
1020 Multicast VPN 1 QOS Vlan
1028 VPN 1 Encap Vlan
1029 VPN 1 Decap Vlan  <<<<<<<<<<<
7600_PE2#

Please note that if both the above highlighted entries do not match, then the entry is not programmed in the hardware. We can verify the correct hardware programming of the VRF using the same command but with a vrf option.

Output on 7600_PE2:
===============
7600_PE2-sp#show platform software multicast ip cmfib vrf ABC 10.11.12.2 239.10.11.12 verbose 
Multicast CEF Entries for VPN#1
(10.11.12.2, 239.10.11.12) 
        MLSVPN:258  (1) PI:1 (1) CR:1 (1) Recirc:1 (1)
        Vlan:1029 AdjPtr:114690 FibRpfNf:1 FibRpfDf:1 FibAddr:0x30066
        rwvlans:1029 rwindex:0x7FFA adjmac:0000.0000.0000 rdt:1 E:0 CAP1:0
        fmt:Mcast l3rwvld:1 DM:1 mtu:1522 rwtype:L3 met2:0x0 met3:0x12
        packets:0000047466040 bytes:000000003227690766
        Starting Offset: 0x0012
           V E L0 C:1025 I:0x02115    <<<<<<<<<

        IOSVPN:256  (1) PI:1 (1) CR:0 (1) Recirc:0 (1)
        Vlan:1029 AdjPtr:114691 FibRpfNf:1 FibRpfDf:1 FibAddr:0x30068
        rwvlans:1029 rwindex:0x7FFA adjmac:0000.0000.0000 rdt:1 E:0 CAP1:0
        fmt:Mcast l3rwvld:0 DM:0 mtu:1518 rwtype:- met2:0x0 met3:0x0
        packets:0000000000000 bytes:000000000000000000

        Annotation-data: [0x5D458BA4]
        A-vlan: 1029 NS-vlan: 0 RP-rpf-vlan: 0
        Anntn flags: [0x100010]  H MT
        MTU: 1500 Retry-count: 0
        Sec-entries count: 1
        Met-handle: 0x4493FED8 New-Met-handle: 0x0
        Met2-handle: NULL
          
        HAL L3-data : [0x49E1831C]
        Flags: 0x4 FIB-index: 0x86B ADJ-index: 0x1C002 NF-addr: 0xFFFFFFFF
        ML3 entry type: 0x0 [(S,G) shortcut]
        Flags: 0xB9400000 Vpn: 258 Rpf: 1029 Rw_index: 0x7FFA
        Adj_mtu: 1514 Met2: 0x0 Met3: 0x12
        V6-data: NULL

        ---Secondary entry [1]---

        HAL L3-data : [0x49E183D0]
        Flags: 0x4 FIB-index: 0x86C ADJ-index: 0x1C003 NF-addr: 0xFFFFFFFF
        ML3 entry type: 0x0 [(S,G) shortcut]
        Flags: 0xA1000000 Vpn: 256 Rpf: 1029 Rw_index: 0x7FFA
        Adj_mtu: 1514 Met2: 0x0 Met3: 0x0
        V6-data: NULL

        ---LSM entries---

Found 2 entries. 1 are mfd entries
7600_PE2-sp#

7600_PE2#sh vlan int usage | in 1025
1025 GigabitEthernet6/1    <<<<<<<<<<<<

Again, both the highlighted entries should match. All of the above logs should also be needed to be collected from the DFC linecard if its involved in Core and customer facing. These outputs should give a fair idea if there is any issue or not.

Hope this information was helpful in understanding mVPN and serves helpful troubleshooting the same on 7600 platform.

Please feel free to reach out to me incase you have any queries.

Happy reading..

Cheers...!!!

Comments (1) -

  • Karan

    8/3/2013 6:55:43 PM |

    This is really interesting piece of information. Thanks for sharing this post.

Comments are closed