Upgrade NetBackup from 8.X.X to 8.1.X On Windows Master fails to upgrade NBDB - error code 1603

Problem

Action ended 12:17:52: INSTALL. Return value 3.
A fatal error occurred during installation on server

Error Message

Install.trace log

11-20-2019,12:13:35 :  Waiting for command: create_nbdb.exe
11-20-2019,12:13:35 :  Running command: "E:\Program Files\Veritas\NetBackup\bin\create_nbdb.exe" -db_server NB_HOSTNAME -db_home "E:\Program Files\Veritas\NetBackupDB" -data "E:\Program Files\Veritas\NetBackupDB\data" -staging "E:\Program Files\Veritas\NetBackupDB\staging" -force .
11-20-2019,12:13:35 :  Waiting for command: create_nbdb.exe
11-20-2019,12:16:56 :  Reading pipe broken.
11-20-2019,12:16:56 :  The process terminated.
11-20-2019,12:16:56 :  Command produced the following output (will display up to 8192 characters):
11-20-2019,12:16:56 :  -------------------------------------------------------------------------->
11-20-2019,12:16:56 :    Database server is NB_chr2pr1bu4
11-20-2019,12:16:56 :    NetBackup database already exists in E:\Program Files\Veritas\NetBackupDB\data\NBDB.db
11-20-2019,12:16:56 :    Starting the NetBackup database server [NB_HOSTNAME].
11-20-2019,12:16:56 :    Creating the NetBackup database files.
11-20-2019,12:16:56 :    The NetBackup database files creation complete.
11-20-2019,12:16:56 :    The NetBackup database upgrade is complete.
11-20-2019,12:16:56 :    Database [NBDB] is not available on server [NB_HOSTNAME].
11-20-2019,12:16:56 :    Exiting from create_nbdb, status[25].
11-20-2019,12:16:56 :  --------------------------------------------------------------------------<
11-20-2019,12:16:56 :  Command returned status 25.
11-20-2019,12:16:56 :  ERROR: NetBackup Database Creation Failed!
11-20-2019,12:16:56 :  Review the log file in NetBackup\logs\nbdb for more information.
11-20-2019,12:16:56 :  CustomAction Deferred_CreateNBDB returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)
11-20-2019,12:16:56 :  Action ended 12:16:56: InstallFinalize. Return value 3.
 
server.log  --
E. 11/20 12:14:33. *** ERROR *** Assertion failed: 200610 (xx.xx.xx.xxx)
Attempting to normalize a non-continued row (0x70:0x3)

Cause

Could not Complete database Validation  During the Upgrade

dbvalid.log --

VALIDATE TABLE "AUTH_MAIN"."AUTH_Access_Token_Details" WITH EXPRESS CHECK
SQL error (-301) -- Internal database error *** ERROR *** Assertion failed: 200610 (xx.xx.xx.xxx)
Attempting to normalize a non-continued row (0x70:0x3) -- transaction rolled back
SQL error (-101) -- Not connected to a database
SQL error (-101) -- Not connected to a database
SQL error (-101) -- Not connected to a database
4 errors reported
SQL Anywhere Validation Utility Version xxxxxxxx
SQL error (-100) -- Database server not found
SQL Anywhere Validation Utility Version xxxxxxxx
SQL error (-100) -- Database server not found

Solution

Rebuild the NBDB database with the following --
----------------------------------------------------
..installpath\Veritas\Netbackup\bin\bpdown -f -v
..installpath\Veritas\Netbackup\bin\bpps (kill any processes)
Services.msc - Stop Veritas Private Branch Exchange
Services.msc - Start NetBackup Relational Database Manager
..installpath\Veritas\Netbackup\bin\nbdb_unload -dbn NBDB -rebuild -verbose
..installpath\Veritas\Netbackup\bin\nbdb_admin -reorganize NBDB
Found xx fragmented records in database.
        Completed 25 % of database tuning.
        Completed 50 % of database tuning.
        Completed 75 % of database tuning.
Database [NBDB] reorganization successful.
..installpath\Veritas\Netbackup\bin\nbdb_admin -validate -full
..installpath\Veritas\Netbackup\bin\bpdown -f -v
..installpath\Veritas\Netbackup\bin\bpps (kill any processes)
Services.msc - Start Veritas Private Branch Exchange
..installpath\Veritas\Netbackup\bin\bpup -f -v

Then rerun the Upgrade