System crashes during an upgrade

System crashes during an upgrade

Article: 100044221
Last Published: 2019-11-29
Ratings: 0 0
Product(s): Appliances

Problem

The system crashes during an upgrade. The crash can happen on either of the two compute nodes. Node A is the node where the command Manage > Software > Install is executed. Node B is the other node in the system.

When the system crashes, the core file is generated under /opt/VRTSnas/core/kernel on the node that crashed.

Error Message

  • Node A crashes

The upgrade process is interrupted and Node A reboots automatically. The upgrade status on Node A displays the following messages:

- [Info] Completed the following pre-upgrade tasks on the node Node_B. The appliance upgrade is 20% complete.
   - Created the pre-upgrade checkpoint
   - Added the service for the post-upgrade tasks
 - [Info] Starting upgrade phase one on the current node Node_A...

The last few lines in the upgrade log on Node A should contain the message:

"upgrade.upgrade_comm.py - DEBUG - "Command is:./access_ctrl.sh stop"
or
"upgrade.upgrade_comm.pyc - DEBUG - Command is:./access_ctrl.sh start"

  • Node B crashes

The upgrade process is interrupted and node B reboots automatically. The upgrade status on Node A displays the following messages:

- [Info] Upgrade phase one is completed on the current node Node_A. The appliance upgrade is 35% complete.
   - Upgraded the Access software
   - Upgraded the appliance software
   - Upgraded the VRTS kernel packages
- [Info] Starting upgrading the node Node_B…"

The last few lines in the upgrade log on Node A should contain the message:
"upgrade.upgrade_comm.py - DEBUG - Command is:ssh root@Node_B 'cd /inst/patch/appliance/installed/7.4.*/scripts;./access_ctrl.sh stop'"

Cause

When the Veritas Virtual Manager (VxVM) service is stopped, the vxio driver is unloaded, which results in the system crash.

Solution

Node A crashes

  1. Log on to Node A. Elevate to the command shell.
  2. Remove the upgrade state file and lock files on Node A:
    • rm /inst/patch/appliance/upgrading/state.info

    • rm /inst/patch/appliance/upgrading/*.lock

  3. Roll back both the nodes manually. For details refer to 100044228 

  4. Upgrade the node again by using the Manage > Software > Install command in the Access Appliance shell menu.

Node B crashes

  1. Reboot Node A.
  2. Manually rollback roll back both  the nodes, please  For details, refer to 100044228
  3. Upgrade the node again. For details, refer to 100044227

Was this content helpful?