NDMP backup fails with 'no mover progress' after backing up significant amount of data

Problem

A backup of a volume or volumes with many millions of files transfers a significant amount of data, then stops until mover progress timeout is reached. While monitoring RAM usage on the media server, memory is maxed out.

Error Message

Error ndmpagent (pid=xxxx) aborting operation - no mover progress

Cause

Insufficient memory on media server. In this example, it was only at 12 GB of memory.

Example:

• The bpbrm process logs normally:
  20:21:36.367 [1656.6104] <2> ConnectionCache::connectAndCache: Acquiring new connection for host backup01.domain.local, query type 78
20:21:36.398 [1656.6104] <2> vnet_pbxConnect: pbxConnectEx Succeeded
20:21:36.398 [1656.6104] <2> logconnections: BPDBM CONNECT FROM 192.168.2.230.62069 TO 192.168.2.25.1556 fd = 592
20:21:36.539 [1656.6104] <2> db_end: Need to collect reply
• Then stops logging at this point.
• The system RAM spikes to 100% utilization and stays there.

Solution

Increase RAM to at least 24 GB to handle multiple NDMP streams and inode metadata for millions of files.

In this example, RAM was increased from 12 GB to 32 GB and the problem went away.