networking-forum.com
Community BlogCommunity Wiki * Register  * Search  * Login
View unanswered postsView active topics

All times are UTC - 6 hours [ DST ]



Post new topic Reply to topic  [ 20 posts ] 
Author Message
 Post subject: iSCSI Latency issue
PostPosted: Fri Aug 10, 2012 11:36 am 
Offline
New Member
New Member

Joined: Fri Aug 10, 2012 11:29 am
Posts: 10
Certs: MCSE, MCTS, MCITP, MSP
Hi Guys,

I have just bought 3 esx hosts to add to my current environment and have bought another 3560G for my core.

I have got a fibre (3x links) trunk between the 2 3560G's(3560A old and 3560B new) my iSCSI SAN is plugged into 3560A and my vmware host has one nic in 3560A and one in 3560B for the iSCSI

If I have both nic's plugged in the iSCSI has loads of latency if i unplug the one in 3560B the latency disapears.

my question is why is this happening and how can i stop it?

also if i plug one nic from the iSCSI SAN into both 3560's will this resolve the latency issue?


I have been stuck with this issue all day and only using one iSCSI isnt an option for redundancy etc.

Thanks
Steve


Top
 Profile  
 
 Post subject: Re: iSCSI Latency issue
PostPosted: Fri Aug 10, 2012 12:34 pm 
Offline
Ultimate Member
Ultimate Member
User avatar

Joined: Thu Jan 13, 2011 5:10 pm
Posts: 985
Location: Leeds, UK
Certs: CCIE R&S #38338, CCNP, CCIP
When you say you have a trunk between the two are you aggregating these links at all?

Also, when you are plugging two NICs into the switches what are you doing on the OS to stop loops? Are you running them in a active/standby manner or some kind of load sharing?

_________________
---
David
CCIE R&S #38338, CCIP, CCNP

http://networkbroadcast.co.uk - My Blog
http://twitter.com/davidrothera


Top
 Profile  
 
 Post subject: Re: iSCSI Latency issue
PostPosted: Fri Aug 10, 2012 1:12 pm 
Online
Post Whore
Post Whore
User avatar

Joined: Fri Nov 13, 2009 5:15 pm
Posts: 1957
Location: Pittsburgh
Certs: CCIE R&S,CCIP,JNCIA,VCP510
So you have 3 new esx hosts.. each has 1 trunk link to switch a one trunk link to switch b and a link for iscsi storage traffic correct? Can you draw this out with a diagram.

_________________
"I will prepare and some day my chance will come." - Abraham Lincoln
http://danielhertzberg.wordpress.com - I blog about networks!


Top
 Profile  
 
 Post subject: Re: iSCSI Latency issue
PostPosted: Mon Aug 13, 2012 3:21 am 
Offline
New Member
New Member

Joined: Fri Aug 10, 2012 11:29 am
Posts: 10
Certs: MCSE, MCTS, MCITP, MSP
davidrothera wrote:
When you say you have a trunk between the two are you aggregating these links at all?

Also, when you are plugging two NICs into the switches what are you doing on the OS to stop loops? Are you running them in a active/standby manner or some kind of load sharing?




I have an LACP Aggregated link on 3 fibre connections between both switches.

I have Vmware running the links as "Route based on source MAC hash" with one link in each switch.

also I don't know if you are aware of how iSCSI works in VMWare but you use two VMK's iSCSI-A which uses nic1 and iSCSI-B which uses nic6 (in the diagram nic6 is disabled to stop performance issues until resolved)

See attached image

the iSCSI San is only in one switch though.


If you would like to see any parts of the switch config I could post some bits.


Attachments:
iscsivmnic.PNG
iscsivmnic.PNG [ 6.2 KiB | Viewed 748 times ]
Top
 Profile  
 
 Post subject: Re: iSCSI Latency issue
PostPosted: Mon Aug 13, 2012 3:52 am 
Offline
New Member
New Member

Joined: Wed May 23, 2012 8:57 am
Posts: 34
I'm pretty sure VMware doesnt support LACP
http://kb.vmware.com/selfservice/micros ... Id=1004048

Have you pinned your iSCSI vmks to individual adapters?
I would also generally use a different subnet per iSCSI path to ensure traffic is sent the way you want.


Top
 Profile  
 
 Post subject: Re: iSCSI Latency issue
PostPosted: Mon Aug 13, 2012 4:06 am 
Offline
New Member
New Member

Joined: Fri Aug 10, 2012 11:29 am
Posts: 10
Certs: MCSE, MCTS, MCITP, MSP
footy wrote:
I'm pretty sure VMware doesnt support LACP
http://kb.vmware.com/selfservice/micros ... Id=1004048

Have you pinned your iSCSI vmks to individual adapters?
I would also generally use a different subnet per iSCSI path to ensure traffic is sent the way you want.



the article that you sent is for LACP between the vmnic and the switch not between two switches.

and yes as above i have setup iscsi-a to vmnic1 and iscsi-b to vmnic6


Top
 Profile  
 
 Post subject: Re: iSCSI Latency issue
PostPosted: Mon Aug 13, 2012 5:00 am 
Offline
New Member
New Member

Joined: Wed May 23, 2012 8:57 am
Posts: 34
marksie1988 wrote:
the article that you sent is for LACP between the vmnic and the switch not between two switches.

and yes as above i have setup iscsi-a to vmnic1 and iscsi-b to vmnic6


Sorry,
Monday morning, head still not quite online yet.
Out of interest have you tested using vmnic6 rather than vmnic1 to try and confirm if the latency is due to traffic going over the new switch or if its when you have the two iscsi vmk's


Top
 Profile  
 
 Post subject: Re: iSCSI Latency issue
PostPosted: Mon Aug 13, 2012 5:12 am 
Offline
New Member
New Member

Joined: Fri Aug 10, 2012 11:29 am
Posts: 10
Certs: MCSE, MCTS, MCITP, MSP
interestingly i have just disabled vmnic1 and tried to get access but i cant.

I cant ping 10.0.0.121 even though its online, the strange thing is if i enable both nic's i can ping 121

WEIRD!!!!

EDIT: I also tried to ping 10.0.0.121 from its directly connected switch.


Top
 Profile  
 
 Post subject: Re: iSCSI Latency issue
PostPosted: Mon Aug 13, 2012 7:00 am 
Offline
New Member
New Member

Joined: Wed May 23, 2012 8:57 am
Posts: 34
did you create the vlans on the new switch?
could you provide a config dump for both switches?
and let us know what ports are use for what if not in desc.


Top
 Profile  
 
 Post subject: Re: iSCSI Latency issue
PostPosted: Mon Aug 13, 2012 7:39 am 
Online
Post Whore
Post Whore
User avatar

Joined: Fri Nov 13, 2009 5:15 pm
Posts: 1957
Location: Pittsburgh
Certs: CCIE R&S,CCIP,JNCIA,VCP510
well for starters vmware allows lacp. Also why is the first storage / iSCSI interface tagged with vlan 10 and the other is untagged?

_________________
"I will prepare and some day my chance will come." - Abraham Lincoln
http://danielhertzberg.wordpress.com - I blog about networks!


Top
 Profile  
 
 Post subject: Re: iSCSI Latency issue
PostPosted: Mon Aug 13, 2012 7:59 am 
Offline
New Member
New Member

Joined: Wed May 23, 2012 8:57 am
Posts: 34
burnyd wrote:
well for starters vmware allows lacp.

Ohh, have you got a link? I wondered when they were going to finally fix this.


Top
 Profile  
 
 Post subject: Re: iSCSI Latency issue
PostPosted: Mon Aug 13, 2012 8:01 am 
Offline
New Member
New Member

Joined: Fri Aug 10, 2012 11:29 am
Posts: 10
Certs: MCSE, MCTS, MCITP, MSP
burnyd wrote:
well for starters vmware allows lacp. Also why is the first storage / iSCSI interface tagged with vlan 10 and the other is untagged?



i had made some config changes to try resolve this issue, they are now both the same please ignore the vlan tag 10 as that isnt like that any more.

attached are cut down switch configs, i have removed passwords and ports that you dont need to see.


Attachments:
switch.zip [3.67 KiB]
Downloaded 21 times
Top
 Profile  
 
 Post subject: Re: iSCSI Latency issue
PostPosted: Mon Aug 13, 2012 8:32 am 
Offline
New Member
New Member

Joined: Wed May 23, 2012 8:57 am
Posts: 34
Can the switches ping each other and the san over vlan 10?
Can both switches the vmk's when using vmnic1, vmnic6 and both.
Have you checked the speed/duplex the ports have negotiated, are these hp servers?


Top
 Profile  
 
 Post subject: Re: iSCSI Latency issue
PostPosted: Mon Aug 13, 2012 8:52 am 
Offline
New Member
New Member

Joined: Fri Aug 10, 2012 11:29 am
Posts: 10
Certs: MCSE, MCTS, MCITP, MSP
footy wrote:
Can the switches ping each other and the san over vlan 10?
Can both switches the vmk's when using vmnic1, vmnic6 and both.
Have you checked the speed/duplex the ports have negotiated, are these hp servers?



1. yes they can no problem at all
2. i can ping both when they are both switched on, when I turn vmnic1 off I cant ping vmnic6 (vmk 121), when i turn vmnic6 off and vmnic1 on i can ping vmk 21 fine.
3. they are dell 720 servers with broadcom nics, they are all running at 1000mb/s


Top
 Profile  
 
 Post subject: Re: iSCSI Latency issue
PostPosted: Mon Aug 13, 2012 8:57 am 
Offline
New Member
New Member

Joined: Wed May 23, 2012 8:57 am
Posts: 34
are you getting any interface errors or drops?


Top
 Profile  
 
 Post subject: Re: iSCSI Latency issue
PostPosted: Mon Aug 13, 2012 9:14 am 
Offline
New Member
New Member

Joined: Fri Aug 10, 2012 11:29 am
Posts: 10
Certs: MCSE, MCTS, MCITP, MSP
sh int gig 0/2
GigabitEthernet0/2 is administratively down, line protocol is down (disabled)
Hardware is Gigabit Ethernet, address is 001d.7143.e982 (bia 001d.7143.e982)
Description: *** ESX01 iSCSI vmnic6 ***
MTU 9000 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto-duplex, Auto-speed, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 03:51:47, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 12
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
3444133 packets input, 799569083 bytes, 0 no buffer
Received 977 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
6981629 packets output, 36800792515 bytes, 0 underruns
0 output errors, 0 collisions, 8 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out


Top
 Profile  
 
 Post subject: Re: iSCSI Latency issue
PostPosted: Tue Aug 14, 2012 6:13 am 
Offline
New Member
New Member

Joined: Fri Aug 10, 2012 11:29 am
Posts: 10
Certs: MCSE, MCTS, MCITP, MSP
I logged this with VM Ware and they have confirmed an issue with a HBA driver / iSCSI storage and possibly not a switch etc.


Top
 Profile  
 
 Post subject: Re: iSCSI Latency issue
PostPosted: Mon Aug 20, 2012 11:32 am 
Offline
New Member
New Member

Joined: Fri Aug 10, 2012 11:29 am
Posts: 10
Certs: MCSE, MCTS, MCITP, MSP
Ok got an update now. Spent ages on the phone with VMWare and they have ruled out vmware from causing this issue.

I plugged in the other new hosts to the new 3560 and every 20 or so pings it drops one, this is causing major latency with iSCSI.

I cant figure this one out as im not that skilled with troubleshooting Cisco layer 3. I think this is a routing issue but could be totally wrong!!!

Any help is appreciated.


Top
 Profile  
 
 Post subject: Re: iSCSI Latency issue
PostPosted: Mon Aug 20, 2012 1:06 pm 
Offline
Ultimate Member
Ultimate Member
User avatar

Joined: Thu Jan 13, 2011 5:10 pm
Posts: 985
Location: Leeds, UK
Certs: CCIE R&S #38338, CCNP, CCIP
Get a TAC case raised? Can't hurt...

_________________
---
David
CCIE R&S #38338, CCIP, CCNP

http://networkbroadcast.co.uk - My Blog
http://twitter.com/davidrothera


Top
 Profile  
 
 Post subject: Re: iSCSI Latency issue
PostPosted: Mon Aug 20, 2012 1:46 pm 
Offline
New Member
New Member

Joined: Fri Aug 10, 2012 11:29 am
Posts: 10
Certs: MCSE, MCTS, MCITP, MSP
I am going to try get a TAC tomorrow but don't think we have a support agreement for this one.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 20 posts ] 

All times are UTC - 6 hours [ DST ]


Who is online

Users browsing this forum: Bing [Bot] and 14 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group