Genie's Tech Blog

Where knowledge has no dimensions

Inter-AS MPLS VPN - Back to Back VRF

Hello All,

Today, We are going to discuss about one of the very common topics on Inter-AS MPLS VPN's i.e. Inter-AS MPLS VPNs with Back to Back VRF. We are already aware of how Intra-AS MPLS VPN's work. This has been discussed deeply in one of our previous posts (Link Below).

http://blog.codergenie.com/blog/page/MPLS-VPN-Basic.aspx

In Inter-AS VPN, Customers connect their sites across AS boundaries. The very first and basic type of Inter-AS deployment is Back to Back VRF. In this approach, the ASBR of both AS's connect each other through multiple sub-interfaces where in each sub-interface segregating different customers. They can all run different protocols but eBGP is the most common one as it can scale as compared to other protocols. In this post we will be discussing about OSPF as the PE-CE protocol between the ASBR of two AS's. Lets consider the below topology:

 

 

In this topology, PE1 and PE2 are in one AS where as PE3 and PE4 are in a different AS. Lets now take a look at the config of all the routers:

Config on CE1:
==========
interface Loopback0
 ip address 100.0.0.1 255.255.255.255
!
interface GigabitEthernet0/1
 ip address 192.168.10.2 255.255.255.0
 duplex auto
 speed auto
 media-type rj45
!         
router ospf 100
 router-id 100.0.0.1
 network 100.0.0.1 0.0.0.0 area 0
 network 192.168.10.2 0.0.0.0 area 0

Config on PE1:
==========
ip vrf ABC
 rd 100:1
 route-target export 100:1
 route-target import 100:1
!
interface Loopback0
 ip address 1.1.1.1 255.255.255.255
!
interface GigabitEthernet0/1
 ip vrf forwarding ABC
 ip address 192.168.10.1 255.255.255.0
 duplex auto
 speed auto
 media-type rj45
!
interface GigabitEthernet0/2
 ip address 12.12.12.1 255.255.255.0
 duplex auto
 speed auto
 media-type rj45
 mpls ip
!
router ospf 1 vrf ABC
 redistribute bgp 100 subnets
 network 192.168.10.1 0.0.0.0 area 0
!
router ospf 100
 router-id 1.1.1.1
 network 1.1.1.1 0.0.0.0 area 0
 network 12.12.12.1 0.0.0.0 area 0
!         
router bgp 100
 bgp router-id 1.1.1.1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 2.2.2.2 remote-as 100
 neighbor 2.2.2.2 update-source Loopback0
 !
 address-family ipv4
 exit-address-family
 !
 address-family vpnv4
  neighbor 2.2.2.2 activate
  neighbor 2.2.2.2 send-community both
 exit-address-family
 !
 address-family ipv4 vrf ABC
  redistribute ospf 1 match internal external 1 external 2
 exit-address-family
!

Config on PE2:
==========
ip vrf ABC
 rd 100:1
 route-target export 100:1
 route-target import 100:1
!
interface Loopback0
 ip address 2.2.2.2 255.255.255.255
!
interface GigabitEthernet0/1
 ip address 12.12.12.2 255.255.255.0
 duplex auto
 speed auto
 media-type rj45
 mpls ip
!
interface GigabitEthernet0/2
 ip vrf forwarding ABC
 ip address 192.168.20.1 255.255.255.0
 duplex auto
 speed auto
 media-type rj45
!
router ospf 1 vrf ABC
 redistribute bgp 100 subnets
 network 192.168.20.1 0.0.0.0 area 0
!
router ospf 100
 router-id 2.2.2.2
 network 2.2.2.2 0.0.0.0 area 0
 network 12.12.12.2 0.0.0.0 area 0
!         
router bgp 100
 bgp router-id 2.2.2.2
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 1.1.1.1 remote-as 100
 neighbor 1.1.1.1 update-source Loopback0
 !
 address-family ipv4
 exit-address-family
 !
 address-family vpnv4
  neighbor 1.1.1.1 activate
  neighbor 1.1.1.1 send-community both
 exit-address-family
 !
 address-family ipv4 vrf ABC
  redistribute ospf 1 match internal external 1 external 2
 exit-address-family
!

Config on PE3:
==========
ip vrf ABC
 rd 200:1
 route-target export 200:1
 route-target import 200:1
!
interface Loopback0
 ip address 3.3.3.3 255.255.255.255
!
interface GigabitEthernet0/1
 ip vrf forwarding ABC
 ip address 192.168.20.2 255.255.255.0
 duplex auto
 speed auto
 media-type rj45
!
interface GigabitEthernet0/2
 ip address 34.34.34.1 255.255.255.0
 duplex auto
 speed auto
 media-type rj45
 mpls ip
!
router ospf 1 vrf ABC
 redistribute bgp 200 subnets
 network 192.168.20.2 0.0.0.0 area 0
!
router ospf 100
 router-id 3.3.3.3
 network 3.3.3.3 0.0.0.0 area 0
 network 34.34.34.1 0.0.0.0 area 0
!         
router bgp 200
 bgp router-id 3.3.3.3
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 4.4.4.4 remote-as 200
 neighbor 4.4.4.4 update-source Loopback0
 !
 address-family ipv4
 exit-address-family
 !
 address-family vpnv4
  neighbor 4.4.4.4 activate
  neighbor 4.4.4.4 send-community both
 exit-address-family
 !
 address-family ipv4 vrf ABC
  redistribute ospf 1 match internal external 1 external 2
 exit-address-family
!

Config on PE4:
==========
ip vrf ABC
 rd 200:1
 route-target export 200:1
 route-target import 200:1
!
interface Loopback0
 ip address 4.4.4.4 255.255.255.255
!    
interface GigabitEthernet0/1
 ip address 34.34.34.2 255.255.255.0
 duplex auto
 speed auto
 media-type rj45
 mpls ip
!
interface GigabitEthernet0/2
 ip vrf forwarding ABC
 ip address 192.168.30.1 255.255.255.0
 duplex auto
 speed auto
 media-type rj45
!
router ospf 1 vrf ABC
 redistribute bgp 200 subnets
 network 192.168.30.1 0.0.0.0 area 0
!
router ospf 100
 router-id 4.4.4.4
 network 4.4.4.4 0.0.0.0 area 0
 network 34.34.34.2 0.0.0.0 area 0
!         
router bgp 200
 bgp router-id 4.4.4.4
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 3.3.3.3 remote-as 200
 neighbor 3.3.3.3 update-source Loopback0
 !
 address-family ipv4
 exit-address-family
 !
 address-family vpnv4
  neighbor 3.3.3.3 activate
  neighbor 3.3.3.3 send-community both
 exit-address-family
 !
 address-family ipv4 vrf ABC
  redistribute ospf 1 match internal external 1 external 2
 exit-address-family
!

Config on CE2:
==========
interface Loopback0
 ip address 200.0.0.1 255.255.255.255
!
interface GigabitEthernet0/1
 ip address 192.168.30.2 255.255.255.0
 duplex auto
 speed auto
 media-type rj45
!         
router ospf 100
 router-id 200.0.0.1
 network 192.168.30.2 0.0.0.0 area 0
 network 200.0.0.1 0.0.0.0 area 0
!

Now the above is a very basic config. We see two MPLS VPN providers one with AS-100 and the other with AS-200. Both PE3 and PE4 are acting as customers to each other as they are sharing routes over VRF for the Customer ABC. With the above basic config, we can see that all the PE-CE protocols are up and also Core routing is up. Lets have a quick look on one of the PE routers:

Output on PE1:
==========
PE1#sh ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override

Gateway of last resort is 10.11.12.1 to network 0.0.0.0

S*    0.0.0.0/0 [254/0] via 10.11.12.1
      1.0.0.0/32 is subnetted, 1 subnets
C        1.1.1.1 is directly connected, Loopback0
      2.0.0.0/32 is subnetted, 1 subnets
O        2.2.2.2 [110/2] via 12.12.12.2, 00:23:59, GigabitEthernet0/2
      10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
C        10.11.12.0/24 is directly connected, GigabitEthernet0/0
S        10.11.12.2/32 [254/0] via 10.11.12.1, GigabitEthernet0/0
L        10.11.12.4/32 is directly connected, GigabitEthernet0/0
      12.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        12.12.12.0/24 is directly connected, GigabitEthernet0/2
L        12.12.12.1/32 is directly connected, GigabitEthernet0/2
PE1#sh mpls for
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    
Label      Label      or Tunnel Id     Switched      interface              
16         No Label   10.11.12.2/32    0             Gi0/0      10.11.12.1  
17         No Label   100.0.0.1/32[V]  0             Gi0/1      192.168.10.2
18         No Label   192.168.10.0/24[V]   \
                                       0             aggregate/ABC    <<<<<<<<<<
19         Pop Label  2.2.2.2/32       0             Gi0/2      12.12.12.2  
PE1#sh mpls ldp nei
    Peer LDP Ident: 2.2.2.2:0; Local LDP Ident 1.1.1.1:0
        TCP connection: 2.2.2.2.23494 - 1.1.1.1.646
        State: Oper; Msgs sent/rcvd: 35/37; Downstream
        Up time: 00:24:08
        LDP discovery sources:
          GigabitEthernet0/2, Src IP addr: 12.12.12.2
        Addresses bound to peer LDP Ident:
          10.11.12.5      12.12.12.2      2.2.2.2         
PE1#sh mpls ldp dis
 Local LDP Identifier:
    1.1.1.1:0
    Discovery Sources:
    Interfaces:
        GigabitEthernet0/2 (ldp): xmit/recv
            LDP Id: 2.2.2.2:0
PE1#
PE1#sh ip bgp vpnv4 all 
BGP table version is 5, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 100:1 (default for vrf ABC)
 *>  100.0.0.1/32     192.168.10.2             2         32768 ?
 *>  192.168.10.0     0.0.0.0                  0         32768 ?
 *>i 192.168.20.0     2.2.2.2                  0    100      0 ?
PE1#

From the above logs we can see that we are learning routes over OSPF withing the core. We can also see label generated for the prefix 2.2.2.2/32 and also an aggregate label generated for the prefix learnt from the customer. We can see the similar outputs on all the PE routers. Lets now take a look at the vrf ABC routes.

Output on PE1:
==========
PE1#sh ip route vrf ABC

Routing Table: ABC
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override

Gateway of last resort is not set

      100.0.0.0/32 is subnetted, 1 subnets
O        100.0.0.1 [110/2] via 192.168.10.2, 00:28:37, GigabitEthernet0/1
      192.168.10.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.10.0/24 is directly connected, GigabitEthernet0/1
L        192.168.10.1/32 is directly connected, GigabitEthernet0/1
B     192.168.20.0/24 [200/0] via 2.2.2.2, 00:25:54
PE1# 

Output on PE2:
==========
PE2#sh ip route vrf ABC

Routing Table: ABC
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override

Gateway of last resort is not set

      100.0.0.0/32 is subnetted, 1 subnets
B        100.0.0.1 [200/2] via 1.1.1.1, 00:27:08
B     192.168.10.0/24 [200/0] via 1.1.1.1, 00:27:08
      192.168.20.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.20.0/24 is directly connected, GigabitEthernet0/2
L        192.168.20.1/32 is directly connected, GigabitEthernet0/2
PE2# 

PE2# sh ip ospf nei

Neighbor ID     Pri   State           Dead Time   Address         Interface
1.1.1.1           1   FULL/DR         00:00:36    12.12.12.1      GigabitEthernet0/1
192.168.20.2      1   FULL/BDR        00:00:36    192.168.20.2    GigabitEthernet0/2
PE2#

From the above logs we notice that we are learning routes on PE2 from the CE1 but we are not learning any routes from PE3 side even though the OSPF neighborship is up. Lets now take a look at the OSPF database on PE2:

Output on PE2:
==========
PE2#sh ip ospf data su 192.168.30.0

            OSPF Router with ID (2.2.2.2) (Process ID 100)

            OSPF Router with ID (192.168.20.1) (Process ID 1)

                Summary Net Link States (Area 0)

  LS age: 1276
  Options: (No TOS-capability, DC, Downward)  <<<<<<<<<<
  LS Type: Summary Links(Network)
  Link State ID: 192.168.30.0 (summary Network Number)
  Advertising Router: 192.168.20.2
  LS Seq Number: 80000001
  Checksum: 0x6A4A
  Length: 28
  Network Mask: /24
        MTID: 0         Metric: 1 

PE2#

From the above output we can see that the route is present in the OSPF database but its having a DN bit set which is causing the route to get installed in the RIB and thus getting exchanged across the MPLS VPN in AS-100. Now the question is why this is happening? The DN bit is set whenever a MPBGP route is redistributed into ospf domain, and the PE router never redistributes a OSPF route with DOWN bit set to MPBGP, therefore essentially preventing the loop. In order to overcome this problem, we need to enable vrf-lite under OSPF.

Config on PE2:
==========
PE2(config)#router ospf 1 vrf ABC
PE2(config-router)#capability vrf-lite 
PE2(config-router)#end

Output on PE2:
==========
PE2#sh ip ospf database external 192.168.30.0

            OSPF Router with ID (2.2.2.2) (Process ID 100)

            OSPF Router with ID (192.168.20.1) (Process ID 1)

                Type-5 AS External Link States

  Routing Bit Set on this LSA in topology Base with MTID 0
  LS age: 80
  Options: (No TOS-capability, DC, Upward)
  LS Type: AS External Link
  Link State ID: 192.168.30.0 (External Network Number )
  Advertising Router: 192.168.20.2
  LS Seq Number: 80000001
  Checksum: 0xE131
  Length: 36
  Network Mask: /24
        Metric Type: 2 (Larger than any link state path)
        MTID: 0 
        Metric: 1 
        Forward Address: 0.0.0.0
        External Route Tag: 3489661128

PE2#

After configuring vrf-lite, we can now see that the routes is now being learnt as a LSA Type-5 and the DN bit is removed. Similar configuration needs to be done no PE3. Once the config is performed, we can now see all the routes getting exchanged between the CE1 and CE2 routers:

Output on CE1:
==========
CE1#sh ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override

Gateway of last resort is 10.11.12.1 to network 0.0.0.0

S*    0.0.0.0/0 [254/0] via 10.11.12.1
      10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
C        10.11.12.0/24 is directly connected, GigabitEthernet0/0
S        10.11.12.2/32 [254/0] via 10.11.12.1, GigabitEthernet0/0
L        10.11.12.3/32 is directly connected, GigabitEthernet0/0
      100.0.0.0/32 is subnetted, 1 subnets
C        100.0.0.1 is directly connected, Loopback0
      192.168.10.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.10.0/24 is directly connected, GigabitEthernet0/1
L        192.168.10.2/32 is directly connected, GigabitEthernet0/1
O IA  192.168.20.0/24 [110/2] via 192.168.10.1, 00:41:41, GigabitEthernet0/1
O E2  192.168.30.0/24 [110/1] via 192.168.10.1, 00:02:23, GigabitEthernet0/1
      200.0.0.0/32 is subnetted, 1 subnets
O E2     200.0.0.1 [110/1] via 192.168.10.1, 00:02:23, GigabitEthernet0/1
CE1# 
CE1#
CE1#ping 200.0.0.1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.0.0.1, timeout is 2 seconds:
Packet sent with a source address of 100.0.0.1 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 11/18/30 ms
CE1#

Lets now also have a look at the BGP and the MPLS forwarding table that will help us understand the flow.

Output on PE1:
==========
PE1#sh ip bgp vpnv4 all 200.0.0.1
BGP routing table entry for 100:1:200.0.0.1/32, version 15
Paths: (1 available, best #1, table ABC)
  Not advertised to any peer
  Refresh Epoch 1
  200
    2.2.2.2 (metric 2) (via default) from 2.2.2.2 (2.2.2.2)
      Origin IGP, metric 1, localpref 100, valid, internal, best
      Extended Community: RT:100:1 OSPF DOMAIN ID:0x0005:0x000000010200 
        OSPF RT:0.0.0.0:5:1 OSPF ROUTER ID:192.168.20.1:0
      mpls labels in/out nolabel/21
      rx pathid: 0, tx pathid: 0x0
PE1#sh mpls for vrf ABC 200.0.0.1
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    
Label      Label      or Tunnel Id     Switched      interface              
None       21         200.0.0.1/32[V]  0             Gi0/2      12.12.12.2  
PE1#sh mpls for vrf ABC 200.0.0.1 det
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    
Label      Label      or Tunnel Id     Switched      interface              
None       21         200.0.0.1/32[V]  0             Gi0/2      12.12.12.2  
        MAC/Encaps=14/18, MRU=1500, Label Stack{21}
        FA163EA0236BFA163E23AD388847 00015000
        VPN route: ABC
        No output feature configured
PE1#

Output on PE2:
==========
PE2#sh mpls for vrf ABC
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    
Label      Label      or Tunnel Id     Switched      interface              
18         No Label   192.168.20.0/24[V]   \
                                       570           aggregate/ABC 
21         No Label   200.0.0.1/32[V]  570           Gi0/2      192.168.20.2
22         No Label   192.168.30.0/24[V]   \
                                       0             Gi0/2      192.168.20.2
PE2#
PE2#sh ip bgp vpnv4 all 200.0.0.1
BGP routing table entry for 100:1:200.0.0.1/32, version 11
Paths: (1 available, best #1, table ABC)
  Advertised to update-groups:
     2         
  Refresh Epoch 1
  200
    192.168.20.2 (via vrf ABC) from 0.0.0.0 (2.2.2.2)
      Origin IGP, metric 1, localpref 100, weight 32768, valid, sourced, best
      Extended Community: RT:100:1 OSPF DOMAIN ID:0x0005:0x000000010200 
        OSPF RT:0.0.0.0:5:1 OSPF ROUTER ID:192.168.20.1:0
      mpls labels in/out 21/nolabel
      rx pathid: 0, tx pathid: 0x0
PE2#

Output on PE3:
==========
PE3#sh ip bgp vpnv4 all 200.0.0.1
BGP routing table entry for 200:1:200.0.0.1/32, version 6
Paths: (1 available, best #1, table ABC)
  Not advertised to any peer
  Refresh Epoch 1
  Local
    4.4.4.4 (metric 2) (via default) from 4.4.4.4 (4.4.4.4)
      Origin incomplete, metric 2, localpref 100, valid, internal, best
      Extended Community: RT:200:1 OSPF DOMAIN ID:0x0005:0x000000010200 
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:192.168.30.1:0
      mpls labels in/out nolabel/19
      rx pathid: 0, tx pathid: 0x0
PE3#

Output on PE4:
==========          
PE4#sh mpls for vrf ABC
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    
Label      Label      or Tunnel Id     Switched      interface              
18         No Label   192.168.30.0/24[V]   \
                                       0             aggregate/ABC 
19         No Label   200.0.0.1/32[V]  570           Gi0/2      192.168.30.2
PE4#

So starting from the PE1, we can see that the outgoing label for prefix 200.0.0.1/32 which is on CE2 is 21 which is the VPN label allocated by PE2 router. At PE2, the label is stripped off and the packet is forwarded as pure IP packet to PE3 where the outgoing label is 19 which is the VPN label allocated by PE4 for the prefix that is learnt from the local CE router CE2. Hope this helps in verifying the flow when you are troubleshooting in a MPLS VPN environment where you have multiple customers.

If you are using any other protocol between the two PE routers, you won't be facing much of a trouble but do remember to configure vrf-lite when having OSPF as the PE-CE protocol between the two PE routers. Also, capability vrf-lite should also be used when you want Multi-VRF support for OSPF. 

Hope this post was helpful.

Cheers...!!!

Comments (1) -

  • anand

    4/10/2014 7:19:23 AM |

    Waiting for another two tough tough options Smile

Comments are closed