Waits and delays for full and empty buffers message recorded in NetBackup logs and also presented in the Activity Monitor for NetBackup 7.1 and later

Problem

Understanding waits and delays for full and empty buffers message produced in NetBackup logs, and also presented in the Activity Monitor for NetBackup 7.1 and later.

Error Message

From bptm log:

08:18:26.087 [5312.6256] <2> write_data: waited for full buffer 24552 times, delayed 1221273 times

From bpbkar log:

08:18:26.072 AM: [5416.6628] <4> BufferManagerLegacySharedMemory::~BufferManagerLegacySharedMemory(): INF - bpbkar waited 64737 times for empty buffer, delayed 66780 times.
 

 

Solution

To calculate how much delay was experienced by bptm, we will need the following two lines from bptm log:
 

08:18:26.087 [5312.6256] <2> write_data: waited for full buffer 24552 times, delayed 1221273 times
00:49:29.035 [5312.6256] <2> io_init: child delay = 10, parent delay = 15 (milliseconds)


 

Delay and Wait counters will help to determine which process has to wait more often: Data Producer or Data Consumer side of the read/write operation.

During a backup, if the client and the media server are the same system, the bpbkar process on the client that reads the data from the storage (disk) and places it into shared memory segments is the Data Producer. The single bptm process that reads this data from the shared memory segments and writes it to the storage (tape or disk), is the Data Consumer.

If the client and media server are different systems, then the Data Producer and Data Consumer are different. The bpbkar process on the client reads the data from the disk and sends it to the network. The bptm Child process on the Media Server then reads that data coming from the client and places the data into shared memory segments on the Media Server, is the Data Producer. Whilst the bptm Parent process that reads the data from the shared memory segments on the media server and writes the data to the destination storage (tape or disk) is the Data Consumer.

The Data Producer needs an empty buffer, while the Data Consumer needs a full buffer. So, the log events above are from the Data Consumer bptm side. 

If the Data Consumer has to wait for a full buffer, it increments the wait and delay counters to indicate that it had to wait for a full buffer. Then the Data Consumer waits the defined number of milliseconds ( which for the Data Consumer is the value from the bptm log output => parent delay = 15 (milliseconds)) before it checks again for a full buffer. If a full buffer is still not available, the Data Consumer increments the delay counter to indicate that it had to wait (delay) again for a full buffer, and waits the prescribed number of milliseconds ( parent delay = 15 (milliseconds)) before rechecking the buffer. The Data Consumer repeats the delay and full buffer check steps until a full buffer is available.

Analysis of the wait and delay counter values that are recorded in the bptm log and also seen in the activity monitor events for the job (in NetBackup 7.1 only and later versions), indicates which process, Data Producer or Data Consumer, has had to wait most often and for how long.
 

Below is a calculation of how much delay is experienced by the bptm process shown in the bptm output example above:

Total delay = how many times bptm was delayed x parent delay (milliseconds) = 1221273 x 0.015 = 18319 secs = 305 mins = over 5 hours

To find out how to solve this problem using the following configuration files, and how these processes (Consumer and Producer) work for backups and restores,  please read NetBackup Planning and Performance Tuning Guide for additional information :

 

SIZE_DATA_BUFFERS
NUMBER_DATA_BUFFERS
PARENT_DELAY
CHILD_DELAY
 

 

 

 

 

Terms of use for this information are found in Legal Notices.

Search

Survey

Did this article answer your question or resolve your issue?

No
Yes

Did this article save you the trouble of contacting technical support?

No
Yes

How can we make this article more helpful?

Email Address (Optional)