How to confirm that Oracle is loading the correct NetBackup Oracle extension library files for use?

When the Oracle RMAN program performs a backup, restore, or catalog maintenance operation using the SBT API, it will utilize a shared object library; libobk* (UNIX/Linux) or orasbt.dll (Windows).  This library is provided by third-party backup software, including the NetBackup Oracle extension. 

Below is a process for confirming if the correct library files are installed and being utilized.  These examples are for UNIX/Linux, but the process is similar for Windows and the details are at the bottom of this document.

1) Shutdown the Oracle instance.  

2) If TNS aliases have or will be used by RMAN to connect to the instance(s) then also shutdown the listener.

3) Confirm all Oracle processes are down.

$ ps -ef | grep -i ora

4) On AIX, also clear the library cache.  This will only work if all Oracle process that utilize the libobk are down and the library use counter has decremented to 0.

$ /usr/sbin/slibclean

 

Note: Confirming that the Oracle process are down is significant!  Once the Oracle instance or listener starts and attempts an SBT API operation, it loads the then current library files from disk into memory.  The running process should unload the library when not needed and then load a new copy when needed, but in rare instances may not.  If that happens, the Oracle instance may remain ignorant of updated copies on disk and continue to use the older copy already loaded into process space.  See 000039293 in the Related Articles section for additional details.
 

5) Capture the last access times on the library files defined by Oracle, NB Oracle, and NB Client.

(
ls -lu $ORACLE_HOME/lib*/libobk*
ls -lu /usr/openv/netbackup/bin/libobk*
ls -lu /usr/openv/lib/libxbsa*
) > /tmp/nbu-lib-access.before.out
 

7) Restart the Oracle instance.

8) Perform a backup, restore or catalog maintenance operation using RMAN.

9) Capture the last access times on the library files again.

(
ls -lu $ORACLE_HOME/lib*/libobk*
ls -lu /usr/openv/netbackup/bin/libobk*
ls -lu /usr/openv/lib/libxbsa*
) > /tmp/nbu-lib-access.after.out

10) Compare the output files from steps 5 and 9.

The access time on one of the NB libobk.* files, should have been updated.  The access time on the libobk.* in one of the lib, lib32, or lib64 sub-directories below $ORACLE_HOME may also have been updated.  

If the access times did not update on the expected library files then either the Oracle instance configuration or the RMAN PARMS statement in the backup/restore/maintenance script specifies an alternate location for SBT_LIBRARY or another libobk* file exists higher on the LD_LIBRARY_PATH (Solaris & Linux), SHLIB_PATH (HP-UX), or LIBPATH (AIX). The DBA will be familiar with the Oracle library load search process and can make the adjustments so that it uses the correct file. 

If the access times updated on an unexpected files, then the DBA will also need to correct the Oracle library load search process.  This may involve specifying SBT_LIBRARY, deleting older libraries that are higher on the search path, or symbolically linking the instance to the NB appropriate libobk.  To build the correct symbolic links, use this script.

$ /usr/openv/netbackup/bin/oracle_link

If the access times did not update on any of the libobk files in the Oracle or NB directories updated, then it is likely that RMAN is connecting to an Oracle instance running on another host instead of on this host.  The DBA should check if the target instance is being accessed via a TNS alias and if the alias is resolving correctly.
 

Recommendation: As an alternative to letting Oracle locate the libobk via search, it is often better to explicitly specify the pathname for the NB libobk when RMAN allocates the channel for backup or restore.  The DBA can make this modification in the backup scripts or restore commands or in the Oracle instance configuration.

 

RMAN_SBT_LIBRARY="/usr/openv/netbackup/bin/libobk.so64"
RMAN_PARMS="PARMS 'SBT_LIBRARY=$RMAN_SBT_LIBRARY'"

...snip...
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE' $RMAN__PARMS;

 

11) After the symbolic links have been corrected or SBT_LIBRARY has been configured, repeat steps 1-10 to ensure the correct library file is in use.

12) If the correct files are being accessed, but the libraries still will not load, then check the NetBackup Database Compatibility matrix to ensure that the platform, architecture (32 or 64 bit), and version of Oracle that is in use is also supported by the version of NetBackup that is installed.
 

13) For the next few days or weeks, enable the dbclient debug log and monitor for these lines.  The build dates for the xbsa library and NetBackup for Oracle should be the same or only a few days different.  If they differ significantly, investigate if a mismatched older library is present and in use by some process.

 

$ cd /usr/openv/netbackup/logs/dbclient
$  egrep 'NetBackup XBSA Interface|NetBackup for Oracle' log.??????
08:30:33.508 [11019] <4> VxBSAInit: Veritas NetBackup XBSA Interface - 7.1  2011020313
Veritas NetBackup for Oracle - Release 7.1 (2011020313)
08:34:13.996 [12828] <4> VxBSAInit: Veritas NetBackup XBSA Interface - 7.1  2011020313
Veritas NetBackup for Oracle - Release 7.1 (2011020313)
09:04:48.405 [26337] <4> VxBSAInit: Veritas NetBackup XBSA Interface - 7.1  2011020313
Veritas NetBackup for Oracle - Release 7.1 (2011020313)


If the build dates are still not correct, then enable the RMAN Channel Trace and inspect the resulting trace file in the user dump destination to see where the library is being loaded from.  E.g. 

$  cat udump/oracle9_ora_21811.trc
...snip...
try loading : libobk.so
Loaded (/app/oracle/lib/libobk.so)

Windows Specific Information: