Problem

There is a low potential for some NetBackup Appliance models 5230 shipped prior to 14-April-2016 where they may encounter a disk drive failure on the software assist Intel ESRT II RAID 1 controller that causes overall system sluggishness or disrupts the appliance boot up process.

This problem on affects the disks that are controlled by and physically connected to the Intel ESRT II RAID 1 controller. NetBackup 5230 Appliance models and disk firmware versions may be susceptible:

• The disk model is Seagate ST1000NM0023
• The disk firmware version is lower than 0006
• The disk firmware version is 0006, and the disk WWN value is less than "0X5000C500855CEAD4"

To determine if your disks require this new firmware version run the Disk Firmware Update Readiness Analyzer Tool before you start disk firmware EEB installation. You can learn more information about the tool and download it from the following link:

000125261 - NetBackup Appliance Disk Firmware Update Readiness Analyzer

Error Message

• Experiencing Overall System Sluggishness
• Disrupts Boot Process showing BUG: soft lockup – CPU#0 stuck for 67s!
• 

Cause

Root cause failure analysis has identified two key issues:

1. The vendor provided Hard Disk Firmware version 0006 initially which was expected to fix the lubrication issue with the disks. However, it was later discovered that there was an additional configuration change required in the update to fully resolve the issue and we received a newer disk firmware version A006 from the vendor with that fix. This update applies only to the disks that were natively running the older 0003 and 0004 disk firmware version as newer stock hard disks coming directly from the vendor on version 0006 are configured correctly to resolve the issue.
2. Default setting for idle A and idle B was changed from 0 (disabled) to 1 (enabled) in the firmware code. However, the actual idle A and idle B settings were not enabled in the initial 0006 firmware version, they were both disabled. In the new version of the firmware version A006 both idle A and idle B are enabled. Native 0006 disks from the vendor have idle A and idle B enabled so they are not affected by the lubrication issue.

Solution

Driver Hotfix Description and list:

OS Driver Version 17.01.2016.0425

Added test unit ready retry timeout

Resolved an issue where erroneous SCSI commands were being rejected by the disk

• SYMC_NBAPP_EEB_ET3877844-2.7.1.0-1.x86_64.rpm
• SYMC_NBAPP_EEB_ET3878738-2.7.2.0-1.x86_64.rpm

Note: The Driver Hotfix was added to NBA v2.7.3 and is included in later releases.

Disk Firmware Hotfix Description and list:

Disk Firmware Version: A006

Provides periodic spans of the unused portion of the drive allowing the read/write head to traverse the span of the disk. This will maintain the lubrication of the drive and resolve the mode of the drive failure identified.

• NBAPP_EEB_ET3917912-2.7.1.0-1.x86_64.rpm
• NBAPP_EEB_ET3917913-2.7.2.0-1.x86_64.rpm
• NBAPP_EEB_ET3917914-2.7.3.0-1.x86_64.rpm
• NBAPP_EEB_ET3917915-3.0.0.0-1.x86_64.rpm
• NBAPP_EEB_ET3958182-3.1.0.0-1.x86_64.rpm
• NBAPP_EEB_ET3958183-3.1.1.0-1.x86_64.rpm

Installation Instructions:

If you are proactively applying these recommended updates please skip to step2 below.

If you are currently experiencing this problem please follow all steps.

Note: You must have the Driver Hotfixes installed on the appliance before you start to install any Disk Firmware Hotfix. Otherwise, the OS disk is taken offline and marked as failed status. This causes the logical driver rebuild. If the rebuild failed, contact the Technical Support.

1. Identify which disk has failed on the ESRT II RAID 1 Controller and remove it. This will allow the appliance to respond properly again and/or boot up fully.

1. Reboot the appliance, press the ESC key to see system post messages and enter the ESRT II RAID 1 Controller Setup Utility by pressing the Ctrl-E Key when prompted.
2. Once invoked, the setup utility main page will be displayed, showing all of the attached disks and their current state.
3. Look for any disk in the failed state, take note of the slot number, exit the utility and power down the appliance.
4. Physically remove that disk based on its slot number, and power on the appliance; a support case will need to be logged to replace that disk.

2. Apply the Driver hotfix (found by accessing the Download Attachments link) that corresponds to your Appliance NetBackup version that handles this particular failure signature and improves platform stability. Please make sure to reboot the appliance after this hotfix has been installed and prior to moving on to step 3.

3. Apply the Disk Firmware hotfix (found by accessing the Download Attachments link) that corresponds to your Appliance NetBackup version to reduce exposure. Please make sure to reboot the appliance after this hotfix has been installed.

Please note: If you downloaded and tried to install the Disk Firmware hotfix and saw the ERROR -> Wrong Disk type message it can be safely ignored.

Appliances that use Seagate models ST1000NM0001 or ST3000NM0001 are not impacted.

This firmware upgrade only applies to the Seagate model ST1000NM023 (5230). This hotfix will only update these disk models that are connected to the appliance software assist ESRT II RAID 1 Controller. There is also a firmware version check that is performed. The following message will be displayed if the disks are already using version 0006:

Internal/External RAID 6 Data disks will not be upgraded or impacted by this hotfix.

It can take some time for the Appliance Hardware Monitor to reflect this firmware version change. If you would like to refresh Hardware Monitor manually to reflect this change please follow these steps:

1. Log into the NetBackup Appliance Console
2. Type Support >Service Restart as-collector

Estimated time to apply these fixes can range from 20 - 80 minutes depending on appliance configuration

Veritas Technologies LLC is aware that the above-mentioned issue is present in the current version(s) of the product(s) mentioned in this article. Veritas is committed to product quality and satisfied customers. The new RAID driver is included in the following release:

• NetBackup Appliances 2.7.3

Please access the following link for download and README information:

https://www.veritas.com/support/en_US/58991.html

Please note, Veritas has gone through an extensive effort to purge all affected disk models with non-native disk firmware version 006 from our spares forward stocking locations. However, in lieu of some partners providing spare disks to their install base, or other spares inventory outside the control of our supply chain, it may become necessary to rerun the Disk Firmware Readiness Analyzer Tool after having replaced a disk to ensure no further updates are necessary. If the Disk Firmware Readiness Analyzer Tool indicates an update is necessary, you can download the EEB version located in this article that corresponds to the NetBackup version running on your appliance. This EEB was designed so that you may execute it as many times as needed without having to perform any uninstallations first.