| « PIX/ASA – Failover, LAN to LAN IPsec VPN, Remote Access VPN | Cisco to Announce New Tech on March 9th » |
| posted in Cisco Networking, Technical | |
| by Steve on January 23rd, 2010 | tags: BGP, challenge, GNS3, ICMP, MP-BGP, MPLS, OSPF, VRF |
I spent some time using GNS3 to build out a 3 router lab focusing on a single customer MPLS VPN. Everything went fine initially, OSPF came up, BGP with route reflectors were fine, LDP neighborships established, and I even saw VRF route advertisements across the network. The problem came when I tried to PING across the network on the VRF from one customer subnet to the other, I got nothing. I spent about an hour troubleshooting, looking up examples, and pulling out my imaginary hair before I figured out the problem. In this blog post I will present the lab as it was originally built for the sake of discussion and leave it up to you to figure out where I went wrong and why exactly it wasn’t working.
Here is the networking drawing as built in GNS3:
Routers R1, R2, and R3 make up the service provider network with R1 being the router reflector and R2 and R3 the RR clients. They are connected via their fast Ethernet interfaces on 10.1.1.0/24 and 10.1.2.0/24. The customer, who is on VRF A, has 1 location on R2 and 1 on R3, 192.168.1.0/24 and 192.168.2.0/24 respectively. OSPF is used as the IGP with BGP/MP-BGP overlaid to carry customer routes. Let’s take a look at their configurations.
R1 Config
interface FastEthernet2/0 ip address 10.1.1.1 255.255.255.0 duplex full speed auto mpls ip ! interface FastEthernet2/1 ip address 10.1.2.1 255.255.255.0 duplex full speed auto mpls ip ! router ospf 1 log-adjacency-changes network 10.1.0.0 0.0.255.255 area 0 ! router bgp 1 no synchronization bgp log-neighbor-changes neighbor 10.1.1.2 remote-as 1 neighbor 10.1.1.2 version 4 neighbor 10.1.1.2 route-reflector-client neighbor 10.1.2.2 remote-as 1 neighbor 10.1.2.2 version 4 neighbor 10.1.2.2 route-reflector-client no auto-summary ! address-family vpnv4 neighbor 10.1.1.2 activate neighbor 10.1.1.2 send-community extended neighbor 10.1.1.2 route-reflector-client neighbor 10.1.2.2 activate neighbor 10.1.2.2 send-community extended neighbor 10.1.2.2 route-reflector-client exit-address-family
R2 Config
ip vrf A rd 1:1 route-target export 1:1 route-target import 1:1 ! interface Loopback0 ip vrf forwarding A ip address 192.168.1.1 255.255.255.0 ! interface FastEthernet2/0 ip address 10.1.1.2 255.255.255.0 duplex full mpls ip ! router ospf 1 log-adjacency-changes network 10.1.1.0 0.0.0.255 area 0 ! router bgp 1 no synchronization bgp log-neighbor-changes neighbor 10.1.1.1 remote-as 1 no auto-summary ! address-family vpnv4 neighbor 10.1.1.1 activate neighbor 10.1.1.1 send-community extended exit-address-family ! address-family ipv4 vrf A redistribute connected no synchronization exit-address-family
R3 Config
ip vrf A rd 1:1 route-target export 1:1 route-target import 1:1 ! interface Loopback0 ip vrf forwarding A ip address 192.168.2.1 255.255.255.0 ! interface FastEthernet2/0 ip address 10.1.2.2 255.255.255.0 duplex full mpls ip ! router ospf 1 log-adjacency-changes network 10.1.2.0 0.0.0.255 area 0 ! router bgp 1 no synchronization bgp log-neighbor-changes neighbor 10.1.2.1 remote-as 1 neighbor 10.1.2.1 version 4 no auto-summary ! address-family vpnv4 neighbor 10.1.2.1 activate neighbor 10.1.2.1 send-community extended exit-address-family ! address-family ipv4 vrf A redistribute connected no synchronization exit-address-family
As you can see, the configs are pretty simple. Basic IP addressing, LDP, OSPF all in area 0, BGP process 1 with the route reflectors, MP-BGP, and finally the VRF creation and it’s associated routing process where the connected loopbacks are redistributed. Simple yes, but let’s take a look at a few things to make sure it’s all set up correctly.
LDP neighborships
R1#sh mpls ldp neigh
Peer LDP Ident: 10.1.1.2:0; Local LDP Ident 10.1.1.1:0
TCP connection: 10.1.1.2.53129 - 10.1.1.1.646
State: Oper; Msgs sent/rcvd: 6/6; Downstream
Up time: 00:01:24
LDP discovery sources:
FastEthernet2/0, Src IP addr: 10.1.1.2
Addresses bound to peer LDP Ident:
10.1.1.2
Peer LDP Ident: 10.1.2.2:0; Local LDP Ident 10.1.1.1:0
TCP connection: 10.1.2.2.12063 - 10.1.1.1.646
State: Oper; Msgs sent/rcvd: 5/5; Downstream
Up time: 00:00:26
LDP discovery sources:
FastEthernet2/1, Src IP addr: 10.1.2.2
Addresses bound to peer LDP Ident:
10.1.2.2
Basic Routing and ICMP
R2#ping 10.1.2.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.1.2.2, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 136/208/248 ms R3#ping 10.1.1.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.1.1.2, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 200/299/396 ms
MP-BGP Neighborships
R1#sh ip bgp vpnv4 all summary BGP router identifier 10.1.1.1, local AS number 1 BGP table version is 3, main routing table version 3 2 network entries using 312 bytes of memory 2 path entries using 136 bytes of memory 2/1 BGP path/bestpath attribute entries using 336 bytes of memory 1 BGP extended community entries using 24 bytes of memory 0 BGP route-map cache entries using 0 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory Bitfield cache entries: current 1 (at peak 1) using 32 bytes of memory BGP using 840 total bytes of memory BGP activity 2/0 prefixes, 2/0 paths, scan interval 15 secs Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 10.1.1.2 4 1 40 39 3 0 0 00:36:18 1 10.1.2.2 4 1 39 38 3 0 0 00:35:31 1
MP-BGP Route Proogation
R2#sh ip route vrf A
Routing Table: A
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route
Gateway of last resort is not set
C 192.168.1.0/24 is directly connected, Loopback0
B 192.168.2.0/24 [200/0] via 10.1.2.2, 00:07:24
R3#sh ip route vrf A
Routing Table: A
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route
Gateway of last resort is not set
B 192.168.1.0/24 [200/0] via 10.1.1.2, 00:07:50
C 192.168.2.0/24 is directly connected, Loopback0
Very cool. Everything seems to be in place so let’s try to PING across the network on the customer’s VRF…
R2#ping vrf A 192.168.2.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.2.1, timeout is 2 seconds: ..... Success rate is 0 percent (0/5)
Doh, we can’t PING across the network. Let’s see if we can PING locally on the customer’s VRF.
R2#ping vrf A 192.168.1.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 8/8/8 ms R3#ping vrf A 192.168.2.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.2.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/8 ms
That works. So why can’t we PING across the SP network on this customer’s VRF? That’s for you to figure out.
I can tell you that this is not a GNS3 issue. There is a very specific problem with the configuration in this network and there is a very specific reason why the configuration is causing the symptom we see above. Please feel free to post your thoughts in the comments, ask questions, etc. If you want a specific show command run, post it up and I will reply with the output. If you know the issue because of past experience or you’re just a networking bad-ass, do me a favor and don’t post it up right away. PM it to me via the forum and I will confirm your solution. I want other folks to have a chance to learn from this post.
For the person who is first to identify the specific config issue AND explain why that is causing the above symptom wins a piece of site swag (I’ll have to recreate some since I deleted all the old logo’d gear) or $20 sent via Paypal.com.
Oh yeah, all I ask in return is that you let folks know about this challenge. Post it in other forums, link to it on your site or blog, send your Cisco-geek buddies an email. I appreciate it and it will help the site grow. Thanks.
Comments have been closed on the blog. Discuss this post on the forum.
The scenario isn’t working since you aren’t using loopbacks for your IBGP VPNV4 connections. In this setup a label pop will accur prior to sending the packet from R2 or 3 to R1. R1 being a P-router has no knowledge of VRF’s and the packet will be dropped. However routes will be advertised between PE through BGP.
You are correct. One person (a double IE) got it before you but he didn’t post the response on the blog. Good work.
ON GNS3..Ip vrf is not supported on loopbacks!!!! so there is no way you can configure either router 2 or 3 as a PE router
You need to use a /32 loopback as the router id for LDP. Without this the PHP is happening one hop too early. This causes R1 to recieve an untagged frame instead of a MPLS frame. R1 doesn’t know what to do with the packet.
MrPaul, you got it, but you were a few minutes too late.
Bummer, Registration killed me.
Also building the .NET in dynamips took a few cycles to double check my answer. Oh well, it was a fun exercise.
Steve,
The issue is with the mask in router 2 & 3. They need to have their mask adjusted to match what is in router 1
R1
router ospf 1
log-adjacency-changes
network 10.1.0.0 0.0.255.255 area 0
R2
router ospf 1
log-adjacency-changes
network 10.1.1.0 0.0.0.255 area 0
R3
router ospf 1
log-adjacency-changes
network 10.1.2.0 0.0.0.255 area 0
No, those reverse masks are good. 10.1.0.0 0.0.255.255 covers both 10.1.1.0/24 and 10.1.2.0/24 on R1 while 10.1.1.0 0.0.0.255 covers 10.1.1.0/24 on R2 and 10.1.2.0 0.0.0.255 covers 10.1.2.0/24 on R3.
And to add to this. The mask on OSPF is just like an ACL. It’s used to match the interface(s) that you want to add to ospf. It has zero impact on the LSAs put in OSPF and does NOT need to match the interface IP/MASK.
network 10.1.1.1 0.0.0.0 area 0 !! adds 10.1.1.1 to OSPF area 0
network 10.1.1.1 0.0.0.255 area 0 !! adds 10.1.1.x to OSPF area 0
network 10.1.1.1 0.255.0.0 area 0 !! adds 10.x.1.1 to OSPF area 0
network 0.0.0.0 255.255.255.255 area 0 !! adds all interfaces to area 0
If you’re so inclined to complicate your configs you’re also free to do bitwise operations too.
BTW, here is what helped me figure it out. In the log of the PE routers, I found this message:
A Google search on “%BGP-4-VPN_NH_IF” took me to this page where it states:
So I rebuilt the lab using loopbacks on the P to PE BGP and MP-BGP sessions and it worked just fine.
I sure thought this one was going to be harder. Good job guys.
Hello..I have an assignment, but the assignment is wrong as we are told to create 2 MPLs vpn but there is only one link from CE1 to PE1 and 2 links from PE2 to seperate branches CE2 and CE3 which we are suppose to implement the vpn to. i configured 1 of the vpn, the routes are showing at each end routers and not in the provider network which is a good thing right..but i can’t ping through and i am trying to trouble shoot but the issue to have another vpn link from CE1 to PE1, do i need to attach another link from CE1 to PE1???? i have the diagram if anybody can brainstorm with me and of course the second vpn is also going to be using IPSec but since the mpls cloud is internal i.e. the company owns d cloud am thinking of implementing ESP transport mode no need of ip encryption just data encryption…thanks