LLT Broadcasting bust every 30 seconds

Problem

It is visible from a tcpdump, that there is a burst of broadcast every 30s originated from each LLT link. According the manual, heartbeats on the hipri links are every half a second, and reduced by 50%  (1 second) on the lowpri links. When these broadcasts are sent through the public lowpri link, may affect in some cases network performance.

The question is: What are those bursts of broadcast, and how to prevent it?

Error Message

To test changed my SAPID of the low pri to 0x1CDE

# cat /etc/llttab
set-node zbox1
set-cluster 20
link qfe2 /dev/qfe:2 - ether 0x2CDE -
link qfe3 /dev/qfe:3 - ether 0x3CDE -
link-lowpri eri0 /dev/eri:0 - ether 0x1CDE -

Did a TCPDUMP
root@zbox1> tcpdump -i eri0 -c 3000 > /tmp/tcpdump_$(hostname)

We see the every second Heartbeat
13:34:11.611132 00:03:ba:24:77:05 (oui Unknown) > zbox2, ethertype Unknown (0x1cde), length 62:
13:34:11.813681 zbox2 > 00:03:ba:24:77:05 (oui Unknown), ethertype Unknown (0x1cde), length 62:
13:34:12.610533 00:03:ba:24:77:05 (oui Unknown) > zbox2, ethertype Unknown (0x1cde), length 62:
13:34:12.813532 zbox2 > 00:03:ba:24:77:05 (oui Unknown), ethertype Unknown (0x1cde), length 62:
13:34:13.610981 00:03:ba:24:77:05 (oui Unknown) > zbox2, ethertype Unknown (0x1cde), length 62:
13:34:13.813561 zbox2 > 00:03:ba:24:77:05 (oui Unknown), ethertype Unknown (0x1cde), length 62:
13:34:14.610566 00:03:ba:24:77:05 (oui Unknown) > zbox2, ethertype Unknown (0x1cde), length 62:
13:34:14.813695 zbox2 > 00:03:ba:24:77:05 (oui Unknown), ethertype Unknown (0x1cde), length 62:
13:34:15.612059 00:03:ba:24:77:05 (oui Unknown) > zbox2, ethertype Unknown (0x1cde), length 62:
13:34:15.813561 zbox2 > 00:03:ba:24:77:05 (oui Unknown), ethertype Unknown (0x1cde), length 62:
13:34:16.610475 00:03:ba:24:77:05 (oui Unknown) > zbox2, ethertype Unknown (0x1cde), length 62:

Then we see that every 30s there is a broadcast of 30 bursts.
So, we send also a bcast out to all remaining CON_WAIT nodes an arp_req every 30s.

# cat /tmp/tcpdump_zbox1 | grep -i ' 00:03:ba:24:77:05 (oui Unknown) > Broadcast'
13:32:46.751657 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751684 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751696 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751773 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751786 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751797 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751808 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751818 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751829 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751840 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751850 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751861 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751893 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751915 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751926 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751937 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751948 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751958 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751969 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751980 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.751991 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.752001 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.752011 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.752022 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.752032 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.752042 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.752052 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.752064 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.752109 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:32:46.752120 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761574 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761594 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761605 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761616 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761627 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761638 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761648 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761659 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761670 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761680 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761691 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761701 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761712 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761733 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761744 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761754 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761764 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761775 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761785 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761796 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761806 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761816 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761827 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761837 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761848 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761859 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761869 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761880 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761893 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:16.761904 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771642 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771666 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771678 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771688 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771699 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771710 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771721 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771731 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771742 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771752 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771763 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771774 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771795 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771807 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771817 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771828 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771838 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771848 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771859 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771869 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771880 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771891 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771902 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771912 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771922 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771932 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771943 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771956 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771967 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:33:46.771977 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812373 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812392 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812402 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812414 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812435 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812446 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812457 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812468 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812478 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812489 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812500 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812510 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812521 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812532 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812542 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812554 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812564 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812574 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812585 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812598 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812608 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812618 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812629 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812639 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812649 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812660 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812670 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812681 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812691 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:
13:34:16.812701 00:03:ba:24:77:05 (oui Unknown) > Broadcast, ethertype Unknown (0x1cde), length 90:

 

Cause

The 30s bursts with 30 broadcast are arp_req to all conwait nodes as per  incident 1942813
Behaviour is seen on both VCS  5.1 and VCS  5.0MP3, but changed in 5.1SP1 with only one arp_req to all nodes.

-> The traffic is of type broadcast and sent every 30 secs on the low-pri net (and highpri)
-> The broadcast is measured one per each possible node which was not yet discovered (CONWAIT)
-> Traffic will continue even if HAD is not running on all nodes, this is GAB/LLT needed only.
 

So basically, there is a burst of arp_req to all possible nodes on 5.1 and 5.0MP3.
 

Solution

Starting in 5.1SP1 every 30s, we send out now only 1 broadcast  to allnodes bcast arp_req instead of the broadcast arp req to all the connwait nodes.
Also in the receiver side only the connwait nodes will respond to this arp_req, and the open nodes actually drop this pkt.
 

Which Solutions to stop apply to stop that broadcast?

-> Move the llt lowpri into an non-public interface.

->  Plumb a second IP in a different network,  to the low-pri link and use LLT  over UDP, this will make sure broadcasts are contained within that network.

-> If running a 2 node cluster, try to exclude nodes 2-31 on the llttab to see if that reduces the number of broadcast

-> Install 5.1SP1 to take advantage of the new feature, which considerable drops the number or broadcast arp_req.
 


Applies To

VCS  5.0MP3 and 5.1

Terms of use for this information are found in Legal Notices.

Search

Survey

Did this article answer your question or resolve your issue?

No
Yes

Did this article save you the trouble of contacting technical support?

No
Yes

How can we make this article more helpful?

Email Address (Optional)