NetBackup™ for MongoDB Administrator's Guide

Consider a configuration with a sharded MongoDB cluster with high availability that contains multiple mongos processes. Before you start a restore and recover operation, only the mongos process on the restore destination for the Config Server Replica Set (CSRS) image should be running.

Manually stop any other mongos processes in the cluster before you start a restore and recover operation.

After recovery reconfigure the mongos services to point to the recovered cluster.

If the mongos process is not shut down on all nodes except one, the additional mongos processes might conflict with the restore operation. This situation causes the data that is restored to be inaccessible with a connection to mongos.

If you do not shut down the mongos processes before you start the restore and recovery, then after recovery you must manually shut down the statemongos processes. Then restart all the recovered mongod and mongos processes under the cluster.

You must start the MongoDB processes with an absolute path to the configuration files. You must use the absolute paths for the certificate files and the CA file as well. You must specify the absolute paths for the CA file, PEM file, and key files as well.

N/A

If the authentication type that was present during backup changes and you run a recovery job that requires a different authentication, the recovery process might fail.

Ensure that the authentication type during recovery remains the same as the type that was used during the backup.

If after you run a backup you then rename the volume group or the logical volume, the subsequent backup may fail.

N/A

During recovery, ensure that you select only one full backup image and the subsequent incremental images that are relevant. If you select more than one image, the recovery may fail as the restored data could be corrupted.

N/A

After your recover the MongoDB cluster, the cluster information for only the restored node is available.

After the recovery process is complete, manually add the secondary nodes to the cluster.

For more information, refer to the following article: add-members-to-the-replica-set

During the backup process, if the MongoDB import operation is running, it can become unresponsive. Avoid the MongoDB import operation during the backup or restore process.

N/A

During the restore process, the message The restore was successfully initiated is displayed, but the restore job does not start.

This issue occurs when you enter the Application server for both the Source client and the Destination client in the web UI.

Ensure that Source client and Destination client are entered correctly. The Source client must be the Application server and the Destination client must be the backup host.

If your environment has DNAT, ensure that the backup host or the restore host and all the MongoDB nodes are in the same private network.

N/A

The NetBackup for MongoDB plug-in does not support the command line bprestore options -w and -print_jobid.

N/A

MongoDB restores are not supported from the backup hosts. All the restore operations for MongoDB must be initiated from the NetBackup primary.

N/A

If your restore job submission does not display the restore job, verify that your destination node has a MongoDB plug-in that is installed on it.

N/A

If you restore the MongoDB database to a non-LVM location and then try to take a backup from this non-LVM location, the backup fails.

Restore the data to an LVM location and then try to take a backup of the restored data.

The NetBackup for MongoDB plug-in does not support hard or soft links in the data path folders. Do not add any hard or any soft links that point to locations in a different logical volume or a non-logical volume.

NetBackup cannot ensure that the data is consistent at the time of backups if you have hard or soft links in the data path folder. During the restore process, the hard or the soft links are created as folders and not links.

N/A

When you cancel a child restore job during the MongoDB restore and recovery process, the thin client (mdbserver) is not removed immediately. The thin client is removed after the next restore operation.

N/A

MongoDB restore fails and displays error 2850.

Consider the following solutions:

After recovery, the MongoDB shard node fails to restart manually and the following error is seen in the MongoDB logs:

NoSuchKey: Missing expected field "configsvrConnectionString"

On the MongoDB shard where the problem occurs, start MongoDB in the maintenance mode and run the following method on the system.version collection in the admin database:

use admin
db.system.version.deleteOne
( { _id: "minOpTimeRecovery" } )

In a restore operation that contains one or more replica sets, replica set members are restored to the replica set that uses the default "cfg.members[#].host" value that rs.config() provides.

If this value was previously updated from the default value after the restore and recover completes, this value may need to be updated to match the original configuration. (For example, from shortname to FQDN.)

Workaround:

Backup jobs fail and the following error codes are displayed:

Ensure that the backup windows for incremental backups are different for the same MongoDB cluster. The backup windows must not overlap each other for incremental backups for the same MongoDB cluster.

Ensure that permissions are in place for the mdbserver location, oplog location, and snapshot mount location. For more information, See Host user requirements.

In a sharded MongoDB cluster environment, a 112 error can indicate that the mongos process is not running on the client that is defined in the backup policy.

An error 112 can also indicate that same hosts names for multiple backup hosts are added to the BigData policy. Use unique host names for multiple backup hosts that are running the backup operations.

After a restore operation, if you try to stop and restart the mongod or mongos services (service mongod stop or service mongod restart), the commands fail.

This error occurs when the mongod or mongos processes are launched as service using the service or systemctl commands and not using a direct command.

Workaround:

Stop the mongod or mongos services using alternative methods. For example, mongod -f /etc/mongod.conf --shutdown or kill <PID>. After stopping the services, you can use the service or systemctl commands again.

Backup operation fails if a command that generates output in .bashrc is added.

Backup fails with error 6646 and displays the following error:

Error: Unable to communicate with the server.

Ensure that no output is generated by .bashrc (echo or any other output generating command). The output should not return STDERR or STDOUT when the shell is non-interactive.

When you select two full backup images and try to restore to a point-in-time image that is between the two full backup images, the latest full backup image is restored.

Workaround:

During the restore operation, do not select more than one full backup image.

For an effective point-in-time recovery, ensure that you run differential incremental backups of shorter duration.

Unable to see the restore job progress in the Recover node.

Workaround:

For a compound restore job that uses a non-primary server as the restore host, look for the job record and status in the Task progress section in the Recover node. Click Refresh to refresh the task list.

Backup fails with the following error:

(6625) The backup host is either unauthorized to complete the operation or it is unable to establish a connection with the application server.

Workaround:

On the server where MongoDB is installed, ensure that PasswordAuthentication is not disabled in /etc/ssh/sshd_config file.

Run the sudo service sshd restart command.

Backup fails with the following error:

(6646) Unable to communicate with the server.

Workaround:

Ensure that the backup host can access the defined port in mongodb.conf file or the default mdbserver_port (11000).

The following error is displayed in the mdbserver logs:

error-sudo: sorry, you must have a tty to run sudo

Workaround:

The nbaapireq_handler log folder is not created on a Flex Container, even you run the mklogdir command.

Workaround:

When a Flex Appliance is upgraded from version 8.1.2 to 8.2 and the Flex media server is used as backup host, the MongoDB plug-in creates the following log directory:

/usr/openv/netbackup/logs/nbaapireq_handler

The snapshot size as described by the free_space_percentage_snapshot parameter must be set according to the MongoDB cluster size and must be large enough. If these criteria are not met, the backup fails and displays the following error:

invalid command parameter (20)

Validate the free_space_percentage_snapshot value with the MongoDB cluster.

Backup fails with the following error:

(13) file read failed for Media

Ensure that the:

The mdb_progress_loglevel parameter is missing from the MongoDB configuration tool.

To modify the mdb_progress_loglevel parameter, update the mongodb.conf file after the MongoDB configuration tool creates it.

For more information, refer to the MongoDB Administrator's Guide.

Snapshots are not deleted and stale mdbserver instances are seen. This scenario might cause Cannot lstat errors during backup and partially successful backups.

Change the configuration settings for the following parameters in the mongodb.conf file:

Set the values such that the stale snapshots and stale instances of mdbserver are cleared before the next full or incremental backup schedule.

If the backup host has NetBackup version earlier than 8.3 and primary and media server have the latest version of NetBackup, the following invalid error codes can be seen for various scenarios:

13302, 13303, 13304, 13305, 13306, 13307, 13308, 13309, 13310, 13311, 13312, 13313, 13314, 13315

Workaround:

Refer to the following list of corresponding actual error codes if you see the invalid error codes for the actual scenarios and recommended actions:

For detailed information and recommended actions, refer to the NetBackup Status Codes Reference Guide.

The Restore button in the NetBackup web UI is disabled for the imported MongoDB backup images.

Workaround:

If you import the images to the same NetBackup primary server that was originally used to back them up, use either of the following methods:

If you import the images to a different NetBackup primary server than the one that was originally used to back them up, use the bprestore command to run the restore operation.

Recovery operation fails on an alternate, sharded MongoDB cluster. The following error is displayed:

Unable to find the configuration parameter. (6661)

This issue occurs during an alternate cluster recovery because the pre-recovery check is unable to find the mongos port for the alternate cluster in the mongodb.conf file. This issue occurs because of the way the MongoDB configuration tool creates the mongodb.conf file when you add the alternate MongoDB cluster details using the Update option from the tool.

Workaround:

Before you start the recovery process, update the mongodb.conf file to separate the alternate cluster from the original cluster.

For example:

Existing mongodb.conf file:

 "application_servers":
   {
    "original.mongodb.cluster.com:26050":
		 {
     "alternate_config_server":
			  [
       {
         "hostname:port": "alt.mongodb.cluster.com:26000",
         "mongos_port": "26001"
       }
      ],
    "mongos_port": "26051"
    }
   }

Suggested update to the mongodb.conf file:

"application_servers": 
   {
    "original.mongodb.cluster.com:26050":
   {
      "mongos_port": "26051"
   },
      "alt.mongodb.cluster.com:26000":
   {
   "mongos_port": "26001"
   }
   }

The MUI tool displays the following error:

Unable to delete configuration.

Recommended action:

Work Around:

Delete the <hostname-port>.conf file manually from /usr/openv/var/global.

In case of certificate-based authentication enabled on MongoDB, a differential incremental backup fails with error 6709: Unable to run the command.

Workaround:

Refer to the mdbserver logs to find the error code and command details. Then perform one of the following actions: