SharePoint 2010 Granular Restore Technology (GRT) backups are failing with error "WRN - unable to setup NBFSD credentials, disabling granular backups"

Problem

SharePoint 2010 'Full Farm' (i.e. non-GRT backups) are completing without error; but the SharePoint 2010 Full 'GRT' backup fail

Error Message

Exact Error Message: Job Details
------------------------------------------
12/17/2013 08:12:53 - Info bpbrm (pid=28398) starting bpbkar on client
12/17/2013 08:12:53 - connected; connect time: 0:00:00
12/17/2013 08:16:18 - Info bpbkar (pid=3968) Backup started
12/17/2013 08:16:18 - Info bpbrm (pid=28398) bptm pid: 28584
12/17/2013 08:16:19 - Info bptm (pid=28584) start
12/17/2013 08:16:19 - Error bpbrm (pid=28398) cannot start nbfsd on Media01 <- MEDIA server?
12/17/2013 08:16:19 - Warning bpbrm (pid=28398) from client sp-few-01: WRN - unable to setup NBFSD credentials, disabling granular backups

Exact Error Message: nbfsd-log.121713
----------------------------------------------------
08:16:19.107 [28588] <2> common_init: nbfsd: starting. @(#) NetBackup_7.5.0.5 Media01 (2013020920, 1387286179 131217.081619)
08:16:19.108 [28588] <2> common_init: nbfsd: unix - SunOS, Media01, 5.10, Generic_147440-19, sun4u
08:16:19.108 [28588] <2> logparams: /usr/openv/netbackup/bin/nbfsd -k -v 0 -p -j 8831492 -B sp-few-01_1387285956 -U BAK -c sp-few-01
08:16:19.118 [28588] <16> my_pmap_dump: can't contact portmapper (12 146)
08:16:19.118 [28588] <16> main: cannot contact portmapper - is portmap service running?
08:16:19.118 [28588] <16> ivfs_shutdown: no image list allocated

Cause

The NFS services (including the NFS 'portmapper' service) were NOT started on the NetBackup media server.

Solution

On the SharePoint Back-End-SQL client server: "sp-sql-01":

Opened a command prompt to <install_path>\NetBackup\bin:

1. Stopped and restarted the Windows NFS Client services:

* Ran > 'nfsadmin client stop'
- Completed without error

* Ran > 'nfsadmin client start'
- Completed without error

2. Checked IF the SharePoint Back-End SQL server could connect to the NFS services on the MEDIA server - it could NOT!

Opened a command prompt to C:\Windows\system32

* Ran > 'rpcinfo -p Media01 **
- rpcinfo: RPC: Unable to send
- rpcinfo: can't connect portmapper

** You can use 'rpcinfo' to list all programs registered with the port mapper; the 'rpcinfo -p [Node]' command lists all programs registered with the port mapper on the specified host.

On the Media server "Media01":

3. Checked if the NFS services on the MEDIA server were running --they were NOT!

Opened a command prompt:
* Ran > "svcs | grep nfs"
- offline 2012 svc:/network/nfs/status:default
- offline 2012 svc:/network/nfs/nlockmgr:default

4. Ran the following command to start the NFS services on the Solaris MEDIA server (Results displayed):

* Ran > svcadm -v enable -r network/nfs/server
svc:/network/nfs/server:default enabled.
svc:/milestone/network enabled.
svc:/network/loopback enabled.
svc:/network/physical enabled.
svc:/network/nfs/nlockmgr enabled.
svc:/network/rpc/bind enabled.
svc:/system/filesystem/minimal enabled.
svc:/system/filesystem/usr enabled.
svc:/system/boot-archive enabled.
svc:/system/filesystem/root enabled.
svc:/system/device/local enabled.
svc:/system/identity:node enabled.
svc:/system/sysidtool:net enabled.
svc:/milestone/single-user:default enabled.
svc:/milestone/devices enabled.
svc:/system/device/fc-fabric enabled.
svc:/system/sysevent enabled.
svc:/system/manifest-import enabled.
svc:/system/filesystem/local:default enabled.
svc:/milestone/single-user enabled.
svc:/system/filesystem/minimal:default enabled.
svc:/system/identity:domain enabled.
svc:/network/nfs/status enabled.
svc:/system/filesystem/local enabled.

5. Checked if the NFS services on the MEDIA server were now running --they WERE!

* Ran > "svcs | grep nfs"
- online 2012 svc:/network/nfs/status:default
- online 2012 svc:/network/nfs/nlockmgr:default

On the SharePoint Back-End-SQL client server: "sp-sql-01":

6. Confirmed the SharePoint Back-End SQL server could connect to the NFS services on the MEDIA server - it COULD!

Opened a command prompt to C:\Windows\system32

* Ran > 'rpcinfo -p Media01

program version protocol port
--------------------------------------------------
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper

On the master server: Master01:

Opened the Java console on the master server

7. Ran a new SharePoint 2010 GRT backup which completed without error.
 

Applies To

SharePoint 2010 Farm Environment:
---------------------------------
Master Server: Master01 - Solaris 10 - running NetBackup 7.5.0.5
Media Server: Media01 - Solaris 10 - running NetBackup 7.5.0.5
Disk Storage: dssu-27 | Disk Type: BasicDisk | Media Server: Media01 |Absolute Path: /dssu-27 | HWM: 90% - LWM: 80%
DSSU Staging Schedule: Frequency: 6 hours | FDSU: Media01-hcart3-robot-tld-0 | FDVP: FIS | Start Window: 3:00 AM -> 8:00 AM , Sun -> Sat
Removable Storage: TLD(0) - Robot: STK L700| Tape Drives: 20 HP Ultrium 3-SCSI (M6BS| L6HS) | Drive Type: hcart3
SharePoint 2010 Client Servers:
-> FEWS: sp-few-01 - Windows 2008 (x64) R2 Enterprise Server SP1 - (Front-end Web server) - running NetBackup 7.5.0.6
-> FEWS: sp-few-02 - Windows 2008 (x64) R2 Enterprise Server SP1 - (Front-end Web server) - running NetBackup 7.5.0.6
-> BESS: sp-sql-01 - Windows 2008 (x64) R2 Enterprise Server SP1 - (Back-end SQL server) - running NetBackup  7.5.0.6

Note: All server names listed above are generic names used only for the examples in this document!