Performance tuning for NetBackup 7.x and 8.x for Solaris 10 and Solaris 11 Project resource allocation

Problem

Error Message

This situation is most commonly identified by the following error message in the NetBackup Job Details.  It occurs when Solaris rejects a request by the bptm or bpdm process for either a shared memory ID or a shared memory segment.

Status Code 89: problems encountered during setup of shared memory

or

10:16:05.024 [20603]<16> setup_dup_shm: Could not open file/usr/openv/netbackup/db/config/shm/clientA_1202256093 to get shared memory information. Errno = 2: No such file or directory
10:16:05.038 [20603]<2> bptm: EXITING with status 12<----------

Cause

Versions of NetBackup prior to 7.0 registered the bpcd and vnetd processes with the Solaris 10 Service Management Facility (SMF) for the purpose of having Solaris listen for inbound connections and then start the appropriate vnetd or bpcd process, with the appropriate project, for each connection.  The bptm or bpdm processes started from bpcd inherited the project.

NetBackup 7.0 changed this behavior by installing the vnetd and bpcd processes as standalone daemons.  Consequently, the project is not associated with vnetd and bpcd and not inherited by the bptm or other processes.  The most common side effect is that the default project does not provide a large enough allocation of shared memory or shared memory identifiers which results in backups failing with status code 89.

Solution

Follow these instructions to register NetBackup processes with Solaris projects created for NetBackup:

To use Solaris projects with NetBackup 8.x and 9.x, it is necessary to customize the environment so that the project is associated with the NetBackup process of interest any time the process is started. The following examples relate to bptm because it is started from vnetd or bpcd. Similar steps should be successful relative to other NetBackup processes.

Note: For Shared Memory settings and Solaris system tuning, see in the NetBackup Planning and Performance Tuning Guide.

The following settings are tuned for a small in-house test environment and are intended as a point of reference only:

• Solaris 11.x master server
• 1 Solaris 11.x media server
• 12 clients

1. Delete any previously created NetBackup project:

$ /usr/sbin/projdel NetBackup

2. Create a Solaris project for NetBackup.  This example creates project 1000, named "NetBackup:"

$ /usr/sbin/projadd -U root -c "NetBackup resource project" -p 1000 NetBackup

3. Execute the following commands to add desired settings for your setup, Message Queues, Shared Memory and Semaphore values:

$ /usr/sbin/projmod -a -K 'project.max-msg-ids=(privileged,1M,deny)' NetBackup
$ /usr/sbin/projmod -a -K 'project.max-sem-ids=(privileged,1M,deny)' NetBackup
$ /usr/sbin/projmod -a -K 'project.max-shm-ids=(privileged,1M,deny)' NetBackup
$ /usr/sbin/projmod -a -K 'project.max-shm-memory=(privileged,32G,deny)' NetBackup
$ /usr/sbin/projmod -a -K 'process.max-file-descriptor=(basic,1M,deny)' NetBackup
$ /usr/sbin/projmod -a -K 'process.max-file-descriptor=(privileged,1M,deny)' NetBackup

4. Use the following command to view the project settings:

$ projects -l NetBackup
NetBackup
      projid : 1000
      comment: "NetBackup resource project"
      users : root
      groups : (none)
      attribs: project.max-msg-ids=(privileged,1M,deny)
               ...
               project.max-shm-memory=(privileged,34359738368,deny)

5. To modify a project attribute, use the projmod command with the -s switch.

$/usr/sbin/projmod -s -K 'project.max-shm-memory=(privileged,10G,deny)' NetBackup
$/usr/sbin/projmod -s -K 'project.max-shm-ids=(privileged,1M,deny)' NetBackup

To verify the change was successful and that the project contains just one entry for the tunable attribute that you specified:

$ projects -l NetBackup | grep max-shm
project.max-shm-memory=(privileged,10737418240,deny)
project.max-shm-ids=(privileged,1048576,deny)

6. NetBackup project settings need to be associated with NetBackup processes.

Ensure that any scripts that [re]start the NetBackup processes, especially vnetd and bpcd, are associated with the project before forking any child processes.

To ensure association at system boot time, modify the /etc/init.d/netbackup script to make this the second line in the script:
MYPID=$$; newtask -p NetBackup -c $MYPID

To ensure association during scripted [re]start of NetBackup, make the same edit to these scripts and any other scripts used to stop or restart netbackup:

• /usr/openv/netbackup/bin/goodies/netbackup
• /usr/openv/netbackup/bin/bp.start_all

Note:  Any custom site-specific changes to a script will need to be reapplied if a future NetBackup release update or hotfix contains an updated copy of that file (script). Also note that this step will need to be redone post any NetBackup upgrade, as the upgrade will lay down new NetBackup scripts removing this and or any custom lines.

7. Ensure that any ad hoc restart of vnetd or bpcd from the command line makes a similar association.  Be sure to confirm that the processes have been stopped, then create a subshell and associate it with the project before restarting the processes.  That way, the project settings apply only to the processes started within the subshell and not to the login shell which is resumed following the exit from the subshell.

$ ps -ef | egrep 'vnetd|bpcd'
    root 13323     1   0   Aug 16 ?           0:05 /usr/openv/netbackup/bin/bpcd -standalone
    root 13320     1   0   Aug 16 ?           0:14 /usr/openv/netbackup/bin/vnetd -standalone

$ /usr/openv/netbackup/bin/bpcd -terminate
$ /usr/openv/netbackup/bin/vnetd -terminate
$ ps -ef | egrep 'vnetd|bpcd'
$

$ sh
MYPID=$$ ; newtask -p NetBackup -c $MYPID
/usr/openv/netbackup/bin/vnetd -standalone
/usr/openv/netbackup/bin/bpcd -standalone
exit

8. Verify the resource limits at any time with the following command:

$ ps -ef | egrep 'vnetd|bpcd'
root 29318 1 0 14:05:31 ? 0:00 vnetd -standalone
root 29315 1 0 14:05:26 ? 0:00 bpcd -standalone

$ /bin/prctl -n project.max-shm-memory 29318
process: 29318: vnetd -standalone
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
project.max-shm-memory
privileged 32GB - deny -
system 16.0EB max deny -

$ ipcs -A
IPC status from as of Mon Dec 1 12:47:11 CST 2014
T ID KEY MODE OWNER GROUP ... PROJECT
Message Queues:
q 44 0x52455051 --rw------- root root ... NetBackup
q 43 0x52455151 --rw------- root root ... NetBackup
T ID KEY MODE OWNER GROUP ... PROJECT
Shared Memory:
m 1275068433 0x5654 --rw-rw-rw- root root ... NetBackup
m 1275068432 0x5643 --rw-rw-rw- root root ... NetBackup
m 16777225 0x88d2d3f0 --rw-rw---- ora dba ... default
T ID KEY MODE OWNER GROUP ... PROJECT
Semaphores:
s 16777324 0x5653 --ra-ra-ra- root root ... NetBackup
s 16777321 0xacb45d9a --ra-r--r-- root root ... NetBackup
s 16777300 0x991c66a4 --ra-ra---- ora dba ... default
s 3 0x6312 --ra-ra-ra- root root ... user.root

Applies To

Solaris 10 and Solaris 11 projects
NetBackup 8.X, 9.x