Hi Guys,

I have just bought 3 esx hosts to add to my current environment and have bought another 3560G for my core.

I have got a fibre (3x links) trunk between the 2 3560G's(3560A old and 3560B new) my iSCSI SAN is plugged into 3560A and my vmware host has one nic in 3560A and one in 3560B for the iSCSI

If I have both nic's plugged in the iSCSI has loads of latency if i unplug the one in 3560B the latency disapears.

my question is why is this happening and how can i stop it?

also if i plug one nic from the iSCSI SAN into both 3560's will this resolve the latency issue?


I have been stuck with this issue all day and only using one iSCSI isnt an option for redundancy etc.

Thanks
Steve

When you say you have a trunk between the two are you aggregating these links at all?

Also, when you are plugging two NICs into the switches what are you doing on the OS to stop loops? Are you running them in a active/standby manner or some kind of load sharing?

_________________
---
David
CCIE R&S #38338, CCIP, CCNP

http://networkbroadcast.co.uk - My Blog
http://twitter.com/davidrothera

So you have 3 new esx hosts.. each has 1 trunk link to switch a one trunk link to switch b and a link for iscsi storage traffic correct? Can you draw this out with a diagram.

_________________
"I will prepare and some day my chance will come." - Abraham Lincoln
http://danielhertzberg.wordpress.com - I blog about networks!

I have an LACP Aggregated link on 3 fibre connections between both switches.

I have Vmware running the links as "Route based on source MAC hash" with one link in each switch.

also I don't know if you are aware of how iSCSI works in VMWare but you use two VMK's iSCSI-A which uses nic1 and iSCSI-B which uses nic6 (in the diagram nic6 is disabled to stop performance issues until resolved)

See attached image

the iSCSI San is only in one switch though.


If you would like to see any parts of the switch config I could post some bits.

I'm pretty sure VMware doesnt support LACP
http://kb.vmware.com/selfservice/micros ... Id=1004048

Have you pinned your iSCSI vmks to individual adapters?
I would also generally use a different subnet per iSCSI path to ensure traffic is sent the way you want.

the article that you sent is for LACP between the vmnic and the switch not between two switches.

and yes as above i have setup iscsi-a to vmnic1 and iscsi-b to vmnic6

Sorry,
Monday morning, head still not quite online yet.
Out of interest have you tested using vmnic6 rather than vmnic1 to try and confirm if the latency is due to traffic going over the new switch or if its when you have the two iscsi vmk's

interestingly i have just disabled vmnic1 and tried to get access but i cant.

I cant ping 10.0.0.121 even though its online, the strange thing is if i enable both nic's i can ping 121

WEIRD!!!!

EDIT: I also tried to ping 10.0.0.121 from its directly connected switch.

did you create the vlans on the new switch?
could you provide a config dump for both switches?
and let us know what ports are use for what if not in desc.

well for starters vmware allows lacp. Also why is the first storage / iSCSI interface tagged with vlan 10 and the other is untagged?

_________________
"I will prepare and some day my chance will come." - Abraham Lincoln
http://danielhertzberg.wordpress.com - I blog about networks!

Ohh, have you got a link? I wondered when they were going to finally fix this.

i had made some config changes to try resolve this issue, they are now both the same please ignore the vlan tag 10 as that isnt like that any more.

attached are cut down switch configs, i have removed passwords and ports that you dont need to see.

Can the switches ping each other and the san over vlan 10?
Can both switches the vmk's when using vmnic1, vmnic6 and both.
Have you checked the speed/duplex the ports have negotiated, are these hp servers?

1. yes they can no problem at all
2. i can ping both when they are both switched on, when I turn vmnic1 off I cant ping vmnic6 (vmk 121), when i turn vmnic6 off and vmnic1 on i can ping vmk 21 fine.
3. they are dell 720 servers with broadcom nics, they are all running at 1000mb/s

are you getting any interface errors or drops?

sh int gig 0/2
GigabitEthernet0/2 is administratively down, line protocol is down (disabled)
Hardware is Gigabit Ethernet, address is 001d.7143.e982 (bia 001d.7143.e982)
Description: *** ESX01 iSCSI vmnic6 ***
MTU 9000 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto-duplex, Auto-speed, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 03:51:47, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 12
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
3444133 packets input, 799569083 bytes, 0 no buffer
Received 977 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
6981629 packets output, 36800792515 bytes, 0 underruns
0 output errors, 0 collisions, 8 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

I logged this with VM Ware and they have confirmed an issue with a HBA driver / iSCSI storage and possibly not a switch etc.

Ok got an update now. Spent ages on the phone with VMWare and they have ruled out vmware from causing this issue.

I plugged in the other new hosts to the new 3560 and every 20 or so pings it drops one, this is causing major latency with iSCSI.

I cant figure this one out as im not that skilled with troubleshooting Cisco layer 3. I think this is a routing issue but could be totally wrong!!!

Any help is appreciated.

Get a TAC case raised? Can't hurt...

_________________
---
David
CCIE R&S #38338, CCIP, CCNP

http://networkbroadcast.co.uk - My Blog
http://twitter.com/davidrothera

I am going to try get a TAC tomorrow but don't think we have a support agreement for this one.