LLT Broadcasting bust every 30 seconds

Article: 100024822
Last Published: 2012-07-28
Ratings: 0 0
Product(s): InfoScale & Storage Foundation

Problem

It is visible from a tcpdump, that there is a burst of broadcast every 30s originated from each LLT link. According the manual, heartbeats on the hipri links are every half a second, and reduced by 50%  (1 second) on the lowpri links. When these broadcasts are sent through the public lowpri link, may affect in some cases network performance.

The question is: What are those bursts of broadcast, and how to prevent it?

Error Message

To test changed my SAPID of the low pri to 0x1CDE

# cat /etc/llttab
set-node zbox1
set-cluster 20
link qfe2 /dev/qfe:2 - ether 0x2CDE -
link qfe3 /dev/qfe:3 - ether 0x3CDE -
link-lowpri eri0 /dev/eri:0 - ether 0x1CDE -

Did a TCPDUMP
root@zbox1> tcpdump -i eri0 -c 3000 > /tmp/tcpdump_$(hostname)

We see the every second Heartbeat
13:34:11.611132 00:03:ba:24:77:05 (oui Unknown) > zbox2, ethertype Unknown (0x1cde), length 62:
13:34:11.813681 zbox2 > 00:03:ba:24:77:05 (oui Unknown), ethertype Unknown (0x1cde), length 62:
13:34:12.610533 00:03:ba:24:77:05 (oui Unknown) > zbox2, ethertype Unknown (0x1cde), length 62:
13:34:12.813532 zbox2 > 00:03:ba:24:77:05 (oui Unknown), ethertype Unknown (0x1cde), length 62:
13:34:13.610981 00:03:ba:24:77:05 (oui Unknown) > zbox2, ethertype Unknown (0x1cde), length 62:
13:34:13.813561 zbox2 > 00:03:ba:24:77:05 (oui Unknown), ethertype Unknown (0x1cde), length 62:
13:34:14.610566 00:03:ba:24:77:05 (oui Unknown) > zbox2, ethertype Unknown (0x1cde), length 62:
13:34:14.813695 zbox2 > 00:03:ba:24:77:05 (oui Unknown), ethertype Unknown (0x1cde), length 62:
13:34:15.612059 00:03:ba:24:77:05 (oui Unknown) > zbox2, ethertype Unknown (0x1cde), length 62:
13:34:15.813561 zbox2 > 00:03:ba:24:77:05 (oui Unknown), ethertype Unknown (0x1cde), length 62:
13:34:16.610475 00:03:ba:24:77:05 (oui Unknown) > zbox2, ethertype Unknown (0x1cde), length 62:

Then we see that every 30s there is a broadcast of 30 bursts.
So, we send also a bcast out to all remaining CON_WAIT nodes an arp_req every 30s.

# cat /tmp/tcpdump_zbox1 | grep -i ' 00:03:ba:24:77:05 (oui Unknown) > Broadcast'
13:32:46.751657 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751684 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751696 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751773 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751786 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751797 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751808 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751818 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751829 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751840 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751850 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751861 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751893 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751915 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751926 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751937 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751948 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751958 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751969 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751980 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751991 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.752001 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.752011 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.752022 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.752032 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.752042 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.752052 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.752064 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.752109 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.752120 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761574 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761594 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761605 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761616 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761627 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761638 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761648 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761659 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761670 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761680 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761691 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761701 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761712 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761733 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761744 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761754 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761764 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761775 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761785 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761796 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761806 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761816 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761827 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761837 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761848 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761859 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761869 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761880 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761893 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761904 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771642 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771666 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771678 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771688 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771699 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771710 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771721 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771731 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771742 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771752 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771763 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771774 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771795 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771807 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771817 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771828 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771838 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771848 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771859 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771869 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771880 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771891 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771902 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771912 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771922 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771932 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771943 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771956 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771967 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771977 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812373 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812392 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812402 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812414 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812435 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812446 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812457 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812468 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812478 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812489 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812500 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812510 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812521 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812532 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812542 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812554 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812564 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812574 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812585 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812598 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812608 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812618 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812629 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812639 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812649 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812660 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812670 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812681 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812691 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812701 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:

 

Cause

The 30s bursts with 30 broadcast are arp_req to all conwait nodes as per  incident 1942813
Behaviour is seen on both VCS  5.1 and VCS  5.0MP3, but changed in 5.1SP1 with only one arp_req to all nodes.

-> The traffic is of type broadcast and sent every 30 secs on the low-pri net (and highpri)
-> The broadcast is measured one per each possible node which was not yet discovered (CONWAIT)
-> Traffic will continue even if HAD is not running on all nodes, this is GAB/LLT needed only.
 

So basically, there is a burst of arp_req to all possible nodes on 5.1 and 5.0MP3.
 

Solution

Starting in 5.1SP1 every 30s, we send out now only 1 broadcast  to allnodes bcast arp_req instead of the broadcast arp req to all the connwait nodes.
Also in the receiver side only the connwait nodes will respond to this arp_req, and the open nodes actually drop this pkt.
 

Which Solutions to stop apply to stop that broadcast?

-> Move the llt lowpri into an non-public interface.

->  Plumb a second IP in a different network,  to the low-pri link and use LLT  over UDP, this will make sure broadcasts are contained within that network.

-> If running a 2 node cluster, try to exclude nodes 2-31 on the llttab to see if that reduces the number of broadcast

-> Install 5.1SP1 to take advantage of the new feature, which considerable drops the number or broadcast arp_req.
 


Applies To

VCS  5.0MP3 and 5.1

Was this content helpful?