Best Practice for maintaining a consistent backup catalog between Oracle RMAN and NetBackup

The overall goal is five-fold.  

• Manage the RMAN Recovery Catalog size by not retaining records about backup pieces any longer then needed.
• Retain backup images in NetBackup until RMAN no longer needs them.
• Manage the NetBackup catalog size and storage media utilization by not retaining backup images that are no longer needed.
• Minimizing the number of times RMAN queries NetBackup for any specific backup piece.
• Minimize the amount of effort it will take NetBackup to locate (or confirm non-existent) the backup image that contains a backup piece of interest to RMAN.

The best approach to managing the catalog synchronization is to use the Oracle and Veritas recommended method.  This method is to set the backup retention primarily as an RMAN attribute, and secondarily as a NetBackup attribute, and then have RMAN delete the backup pieces which are obsolete, but not expired, from NetBackup.  The procedures to do this are documented in the Oracle Recovery Manager Guides for Oracle versions nine and later (www.oracle.com/technology/documentation/index.html). An outline of those procedures is shown below, but refer to the NetBackup and Oracle documentation for a complete description.

1. Set RMAN retention for the number or duration of backup sets.  This is to keep the pieces in the RMAN catalog as long as needed.  If there is no RMAN catalog then use SQL to set an appropriate value for "control_file_record_keep_time".  The minimum appropriate time would be the required backup retention time plus the maximum time between catalog maintenance operations.
2. Set the NetBackup retention for Oracle backups images to be longer than the RMAN retention.  This allows NetBackup to retain the backup images until RMAN both determines the images to be obsolete and deletes them from both catalogs.  Thus avoiding exhaustive searches for pieces names that are known to RMAN but no longer known to NetBackup.  If RMAN backups of the Oracle instance utilize more than one NetBackup policy and/or schedule, be sure each schedule is configured with the appropriate retention.  Use an infinite retention only if the backups images either, will be expired manually at some future date, or must be retained forever.  Unnecessary bloat, accumulated over time, will slow some searches.
3. Ensure that the FORMAT specified for all RMAN backup piece names, except for AUTOBACKUP CONTROLFILE, ends with a '_%t' as documented in the NetBackup for Oracle Administrator's Guide.  Both the trailing '%t' and the default format for AUTOBACKUP CONTROLFILE are parse-able for the Oracle backup time, and allow NetBackup to search a narrow time range for non-expired piece names.  (See Note-1 and Note-2 below.)
4. On a regular and frequent basis, run the RMAN "delete obsolete" command to expire obsolete images from both the RMAN catalog & control file and also from NetBackup.  This minimizes the size of both catalogs as soon as possible, and also releases the storage media used for those backups.  (See Note-3 below.)
5. An RMAN "crosscheck" command is typically not necessary.  If used, run it after obsolete pieces have been deleted, to minimize the number of pieces that need to be checked.  Use the command infrequently to avoid regularly and repeatedly querying the master server for the same pieces, especially if NetBackup retention is shorter than RMAN retention.  That condition will result in exhaustive searches which consume more CPU and I/O resources on the master.  In the latter case, the RMAN "delete expired backup" command should be run afterwards to make sure the same pieces are not queried, with the same exhaustive search and results, by the next crosscheck.  (See also Note-4 below.)
6. Stagger the initiation of RMAN catalog maintenance operations to limit the number of concurrent crosscheck or deletion requests to the NetBackup master server by the clients.  Especially if multiple channels are being allocated for multiple databases that backup to the same policy client.  In addition, ideally catalog maintenance would not overlap with busy Oracle backup windows for the same client, since backups performs similar queries.  Overlap with image duplication and replication operations is not an issue.
7. Where possible, avoid the creation of many small RMAN backup pieces, especially for archivelog and controlfile backups.  Modern network and storage devices can transfer gigabits of data per second, so backup piece sizes should be in the tens/hundreds/thousands of gigabytes to minimize the ratio of job overhead delay to actual backup transfer time.  As an additional side effect, fewer backup piece names will be present and need to be tracked, queried, and expired.  Carefully consider the RMAN configuration for the backups; fewer channels, transporting larger backup sets and piece, filled from a larger numbers of files will result in fewer pieces.  This is most applicable to archivelog backups, but may be useful for database files which have poor deduplication ratios.

Additional Considerations

8. Keep the clocks on the master server and client hosts set accurately, and with appropriate time zones, so that the first query with time range +/- 24 hours is successful.
9. Ensure that 'bpclntcmd -pn' completes quickly, ideally sub-second, when run on each Oracle client host.  If not, troubleshoot the network connectivity between and name services on both hosts until any delays have been identified and resolved.  Each request from RMAN to NetBackup for Oracle is going to cause one or more similar connections.  RMAN will often be making tens, hundreds, or even thousands of requests so the cumulative delays can be significant.
10. Delete inactive policies that are unlikely to ever be returned to service.  Otherwise NetBackup will be repeatedly reading them to build policy client lists.
11. Ensure all policy clients are the 'right' hostnames for the client.  The master server and client should generally identify the client host by the same hostname.  (See Note-5 below.)
12. Ensure all policy client hostnames consistently resolve quickly on the master server.  This allows the master server to quickly find network aliases which might be in use by the client.  (See Note-5 below.)
13. Are other resource (CPU, memory, I/O, etc.) intensive operations occurring on the master server host at the same time; many hundreds or thousands of processes and network connections for other active backup/restore jobs, catalog backup, master running media server processes, file system maintenance, etc.?

If it is not possible to use the RMAN retention policy and "delete obsolete", then use these steps with RMAN "crosscheck" and "delete expired" commands.

• Regularly review the retention of backup images in NetBackup and ensure they are not excessively long.
• Regularly use the NetBackup bpexpdate command to expire any images that can be delete sooner than their initial retention setting.
• Regularly review steps 5 and 6 above to ensure the scheduling of RMAN crosscheck and delete expired is occurring regularly, but not so often as to perform redundant operations before additional images are expired from within NetBackup.
• Regularly review recent backups to ensure steps 3 and 7 above are being followed.
• Regularly confirm step 8-12 above.

Note-1: This does not benefit queries for expired images because they will still search the entire time span available when the piece is not found in the narrow time span.  That is because a piece name might be formatted to include some other string of digits in the location where an Oracle time is expected, so all backup images must be searched to ensure the piece name is truly not present.  

Note-2: In February of 2019, the Oracle time rolled over from 9 digits to 10 digits.  NetBackup for Oracle client versions prior to NetBackup 8.2 will not recognize the new time format without having a fix applied.  See the related article for details.

Note-3: For proxy backups where a single NetBackup image contains multiple RMAN backup pieces, an RMAN request to delete a piece will not result in the deletion/expiration of the image because other pieces still depend upon the image being present.  If RMAN later re-catalogs the piece, NetBackup will be unaware.  Hence this images will persist until the NetBackup retention expires.

Note-4: If performing crosscheck operations, be careful to ensure that the Oracle server process performing the sbtinfo API calls is running as a user or group which matches the backup piece ownership and access permissions.   Otherwise, NetBackup will report the piece 'not found' and RMAN will mark it as expired, and then at a later time potentially delete it's record of the supposedly expired piece.  The same is true if the crosscheck uses a browse client that does not match the policy client used at time of backup, see also Note-5 below.  A similar result also occurs if NB_ORA_POLICY is configured during the crosscheck (typically it is not) and does not match the policy used for some of the backups.  If NetBackup reports that a backup piece is not found, due to incorrect query criteria, Oracle 10.2+ can re-query NetBackup (with correct criteria) and re-catalog the piece using the RMAN "catalog device type 'sbt_tape' backuppiece '<piecename>'" command.

Note-5: Through normal business flow (equipment replacement or relocation, mergers & acquisitions, etc.) the hostname by which a host is known may change.  If the host is backed up using a new policy client name, then the new backup images will be stored under the new name, and a crosscheck of both old and new images will only be successful for one group of images (whichever is currently configured as the NB_ORA_CLIENT or CLIENT_NAME).  To avoid such failures, the new hostname can be made a client alias of the original hostname, so the crosscheck can be performed using either client name.  Ideally, this would be done before any backups are taken using the new hostname, but it is also possible to merge images taken using two client names.  See the related article for details.