Is it necessary to shutdown Oracle when upgrading third-party backup software such as the NetBackup for Oracle extension?

Problem

Oracle database instances are typically in use 24 x 7 which makes it challenging to stop and restart the Oracle processes just to upgrade the third-party backup software.  

The architecture of the SBT API and the Oracle Recovery Manager (RMAN) is such that the third-party extension is implemented as a shared object library (e.g. libobk.so.1 or orasbt.dll = NetBackup for Oracle).  The library is loaded into an Oracle process to affect the backup or restore. In legacy versions, the library depended on additional library files (e.g. libxbsa.so.1 or *xbsa*.dll = NetBackup Client) so it was necessary to make sure compatible library files are loaded into the Oracle process.  

The challenges that are imposed by the operating system and Oracle are as follows.

1. Successfully install the new library files for NetBackup Client and NetBackup for Oracle onto the host file system.

2. Ensure that any Oracle process that performs a backup or restore loads the new library files from the file system instead of using older library files that are already mapped into the process space.

Solution

It is not necessary to shut down Oracle provided that concurrent and future operations on the client host do not conflict with challenges #1 and #2 above.   However, shutting down the Oracle instance and listener processes on the host guarantees that there won’t be any conflicts and that any future backup images will have integrity and can be successfully restored.

If shutting down the Oracle instance and listener processes long enough to do the install is not possible, the following steps may allow for an upgrade depending on operating system and Oracle constraints.   Please be aware that backup images created using a mix of old and new libraries may not be usable upon restore.

Before the upgrade:

1. Ensure that the Oracle and RMAN versions and backup/restore procedures used will dynamically load the SBT API for each backup and dynamically unload the libraries after each backup is complete.   This is the default configuration for Oracle 9i and newer, but some configurations have been encountered where this is not the case.

2. Review the NetBackup dbclient debug logs from the past several days to confirm the build dates for the currently utilized NetBackup for Oracle and NetBackup Client (XBSA) libraries.   These log entries are visible at VERBOSE = 4, e.g.
 
... NetBackup XBSA Interface - x.x   2009050105
... NetBackup for Oracle - Release x.x (2009050106)

UNIX/Linux log location:  /usr/openv/netbackup/logs/dbclient
Windows log location: <install_dir>\NetBackup\logs\dbclient

Ignore the last two digits of the build time; the hour of the build is not significant, only the date.  There should only be one build date for each component.  The library files for both components should be built on the same date. 

If there is more than one date, then an mix of old and new libraries is already in use and current backups may not have integrity.  

Save the information for later review during steps 7 and 8.    
   

3. Ensure that the library files on disk are not in use and can be overwritten.   If these files are in use, the operating system (OS) may not allow them to be overwritten.  

In this example from HP-UX, the PIDs of any conflicting processes are shown after the file names.    A 32-bit Oracle instance is currently using some libraries and should be shutdown before continuing with step 4.

$ fuser /usr/openv/lib/libxbsa*

libxbsa.so: 9246o

libxbsa64.so:

libxbsaMT.so: 9246o

libxbsaMT64.so:

$ fuser /usr/openv/netbackup/bin/libobk*

libobk.so.1:  9246o

libobk.so64.1:

$ fuser /usr/openv/netbackup/bin/libsapora*

libsapora.so64.1:

4. Perform the upgrade according to the upgrade installation documentation.

5. On AIX, attempt to clear the library cache.   If all Oracle processes are shutdown, this will clear related libraries from the OS cache and ensure that the new libraries are reloaded from disk after the upgrade.  Old libraries that are in use by currently running Oracle processes will remain in the cache, remain in use, and may cause problems.

$ slibclean
 

6. Clear the dbclient debug log for the current date, e.g.

$ cd /usr/openv/netbackup/logs/dbclient
$ mv log.080810 log.080810.old

Windows: 

> cd <install_dir>\NetBackup\logs\dbclient

> rename 080810.log 080810.log.old

7. After the next backup, review the dbclient debug log for the current date. 

Confirm that the build dates from the upgraded library files are visible and newer than those captured in step 2.  

8. It is strongly recommended to review the dbclient logs daily until a full cycle of all backups is completed.  

If a mix of old and new build dates are visible, then there is one or more Oracle processes that have copies of the old libraries mapped into process space.   It is necessary to stop and restart those processes to ensure that the most recently installed libraries are used.

Note: In current versions of NetBackup, libobk and libxbsa are statically linked for compatibility with Oracle 10 R2 and an incompatible mix of binaries is not possible.  However, the procedure above should still be followed to ensure that all backups are using the most recently installed library.
 

Applies To

Oracle 9i, 10g, 11g, 12c