System crashes during an upgrade

Problem

The system crashes during an upgrade. The crash can happen on either of the two compute nodes. Node A is the node where the command Manage > Software > Install is executed. Node B is the other node in the system.

When the system crashes, the core file is generated under /opt/VRTSnas/core/kernel on the node that crashed.

Error Message

• Node A crashes

The upgrade process is interrupted and Node A reboots automatically. The upgrade status on Node A displays the following messages:

- [Info] Completed the following pre-upgrade tasks on the node Node_B. The appliance upgrade is 20% complete.
   - Created the pre-upgrade checkpoint
   - Added the service for the post-upgrade tasks
 - [Info] Starting upgrade phase one on the current node Node_A...

The last few lines in the upgrade log on Node A should contain the message:

"upgrade.upgrade_comm.py - DEBUG - "Command is:./access_ctrl.sh stop"
or
"upgrade.upgrade_comm.pyc - DEBUG - Command is:./access_ctrl.sh start"

• Node B crashes

The upgrade process is interrupted and node B reboots automatically. The upgrade status on Node A displays the following messages:

- [Info] Upgrade phase one is completed on the current node Node_A. The appliance upgrade is 35% complete.
   - Upgraded the Access software
   - Upgraded the appliance software
   - Upgraded the VRTS kernel packages
- [Info] Starting upgrading the node Node_B…"

The last few lines in the upgrade log on Node A should contain the message:
"upgrade.upgrade_comm.py - DEBUG - Command is:ssh root@Node_B 'cd /inst/patch/appliance/installed/7.4.*/scripts;./access_ctrl.sh stop'"

Cause

When the Veritas Virtual Manager (VxVM) service is stopped, the vxio driver is unloaded, which results in the system crash.

Solution

Node A crashes

1. Log on to Node A. Elevate to the command shell.
2. Remove the upgrade state file and lock files on Node A:

   • rm /inst/patch/appliance/upgrading/state.info

   • rm /inst/patch/appliance/upgrading/*.lock

3. Roll back both the nodes manually. For details refer to 100044228 

4. Upgrade the node again by using the Manage > Software > Install command in the Access Appliance shell menu.

Node B crashes

1. Reboot Node A.
2. Manually rollback roll back both  the nodes, please  For details, refer to 100044228
3. Upgrade the node again. For details, refer to 100044227