How to use NBPerfchk to troubleshoot performance issues related to disk io and network transfer rates

NBPerfchk.exe is a standalone tool that can be used to find what the base disk performance is on volumes used for backup disk based devices. It can also be used to find what the network throughput is to a disk volume from a remote disk volume.

Download the zip file attached to this article and extract it to a meaningfully named folder in a path of your choosing.

nbperfchk -i inputpath -o outputpath [-s filesize] [-syncend] [-bs buffersize] [-v]

Run from Command prompt and run as administrator.

To use when troubleshooting reported performance issues with backups, duplicate or restore jobs jobs using disk storage devices. As well as for network performance.
If the test using the tool shows same performance as the Backup software then the performance limitation will be in the environment and not with the product.

Usage: nbperfchk <options>

  [-bs {buffer-size}]        - buffer and I/O size; default 65536
  [-n  {number-of-buffers}]  - default 255
  [-s  {SIZE}]               - copies this much data; default all
  [-i  {INPUT}]              - input; default stdin
  [-o  {OUTPUT}]             - output; default stdout
  [-directio]                - use directio (unbuffered)
  [-syncend]                 - sync/flush output file at end
  [-ri {INTERVAL}]           - report interval in seconds; default 3
  [-rc]                      - report compact - no linefeeds
  [-rp]                      - report probe; display buffer states
  [-nr]                      - no progress reporting
  [-q]                       - quiet - no reporting
  [-v]                       - verify data at client

sizes can be followed by 'k', 'm', or 'g'

{INPUT} is one of {file-path}, zero:, random:, seq: , or tcp:{HOST}:{PORT}, if the client need to verify data, seq:  is required
{OUTPUT} is one of {file-path}, null:, or tcp:{HOST}:{PORT}

Note:

Veritas recommends that you provide the output from using this tool when you create a new support case for performance issues so the Veritas technical support staff member can get to know this upfront. Veritas recommends a minimum disk performance level of 130 MB/sec for deduplicated read and write operations.
 

Disk testing  - write
The tool will test the read speed and the write speed volume for the backup disk device used or to be used
Start with the write test as that will create a file that can later be used by the read test
The below example parameters will write a file of all zeroes to named file path and it will be 64GB in size. The folder path needs to exist, the tool will create the test file.

nbperfchk -i zero: -o D:\TEST\file.test -s 64g -syncend

The file size needs to be larger than the RAM on the server to get past reads/writes from memory. As such you will see that it will start with a high performance output and then tail of to what the real disk performance is. The  -syncend parameter will flush the buffers at the end of the write test and writes all the data to disk.
 
C:\BEPERFCHECK>nbperfchk -i zero: -o D:\TEST\file.test -s 64g -syncend
2338 MB @ 779.7 MB/sec, 2339 MB @ 779.8 MB/sec
5161 MB @ 860.2 MB/sec, 2821 MB @ 940.6 MB/sec
8094 MB @ 899.5 MB/sec, 2933 MB @ 977.9 MB/sec
9193 MB @ 766.2 MB/sec, 1097 MB @ 365.8 MB/sec
9542 MB @ 636.2 MB/sec, 349 MB @ 116.5 MB/sec
9656 MB @ 536.5 MB/sec, 113 MB @ 37.7 MB/sec
9686 MB @ 461.3 MB/sec, 30 MB @ 10.0 MB/sec
9780 MB @ 407.5 MB/sec, 94 MB @ 31.4 MB/sec
10011 MB @ 370.8 MB/sec, 230 MB @ 76.9 MB/sec

The left two columns are the total data written and the average speed as it was written. The two right columns are the last data written and the average write speed of the last 3 seconds.
The volume should be idle in regards to backup/restore operations while running this tool. 
If you see fluctuations in the right hand column MB/sec like shown above throughout the test will indicate that this is a very poor performing or busy volume and disk queue length will be more than 3. Use Windows Resource Monitor by clicking on the Disk tab and expand the Storage section below to see the queue length on the volume being tested. Or use Windows Performance Monitor and add for Physical disk Avg. Disk Read Queue Length and Avg. Disk Write Queue Length, add the volume by letter below before clicking Add and OK. Set the Graph scale in the properties to 1000 (1000=queue length of 10).
As the test runs the performance output will tail out and have little fluctuation. The last line after the sync is indicative of the volumes write speed.

65176 MB @ 240.5 MB/sec, 848 MB @ 282.9 MB/sec
Starting file sync to flush buffers...
65536 MB @ 239.2 MB/sec, 358 MB @ 119.5 MB/sec
Sync completed within 7 seconds
65536 MB @ 234.9 MB/sec

In this example the write performance was 239MB/sec.

Disk testing - read

Now that a 64GB has been created it can now be used for read performance testing.
nbperfchk -i inputpath -o NUL -bs buffersize

 

For Linux environments use  -o /dev/null for the output location to avoid writing a file on the local file system.

So as an example on Windows using the file created by the write test above using first a 64K buffer size then a 1024K buffer size
nbperfchk -i D:\TEST\file.test -bs 64k -o NUL

 

D:\BEPERFCHECK>nbperfchk -i D:\TEST\file.test -bs 64k -o NUL
609 MB @ 203.1 MB/sec, 609 MB @ 203.1 MB/sec
1246 MB @ 207.7 MB/sec, 636 MB @ 212.3 MB/sec
1810 MB @ 201.2 MB/sec, 564 MB @ 188.1 MB/sec
2427 MB @ 202.3 MB/sec, 616 MB @ 205.5 MB/sec
|
|
59811 MB @ 171.9 MB/sec, 686 MB @ 228.8 MB/sec
60456 MB @ 172.2 MB/sec, 644 MB @ 214.9 MB/sec
60954 MB @ 172.2 MB/sec, 497 MB @ 166.0 MB/sec
60954 MB @ 172.7 MB/sec

In this example read performance with a 64K buffer size is 172 MB/sec
D:\BEPERFCHECK>nbperfchk -i D:\TEST\file.test -bs 1024k -o NUL
1141 MB @ 380.3 MB/sec, 1140 MB @ 380.0 MB/sec
2182 MB @ 363.7 MB/sec, 1041 MB @ 347.0 MB/sec
3318 MB @ 368.8 MB/sec, 1136 MB @ 378.7 MB/sec
4438 MB @ 369.8 MB/sec, 1120 MB @ 373.3 MB/sec
|
|
58992 MB @ 299.5 MB/sec, 1193 MB @ 398.0 MB/sec
60264 MB @ 301.3 MB/sec, 1271 MB @ 423.7 MB/sec
60954 MB @ 300.3 MB/sec, 690 MB @ 230.3 MB/sec
60954 MB @ 303.3 MB/sec

In this example read performance with a 1024K buffer size is 303 MB/sec.
Both these two examples show good performance.

If you see, as in the test output above, that a 1024K buffer size has significantly better performance than 64K will indicate that changing the ReadBufferSize for the Data store read buffer size in the contentrouter.cfg file ( <dedup root path>\etc\puredisk\contentrouter.cfg) will improve restore and duplicate job performance from a deduplication storage device. Test with different buffer sizes (128k, 256k etc) to see what performs best but in most cases that will be 1024K (1MB). From Backup Exec version 16, 1024K is the default value.

Always make a copy of the .cfg before any edit, and never use Notepad for editing (use Wordpad, Textpad etc). Backup Exec services including for deduplication, needs to be restarted after making the change.

As for the write test the disk queue length is equally important for read. So if you see disk queue length much higher than 3 and read of less than 150MB/sec Backup Exec will not be able to perform at its optimum and are limited by the environment it has to operate in.


Network test
This tool can also be used to test data transfer to disk over the network.
First create the reader. This will in most cases be the media server. The target that will receive the data. Use any available TCP port.
nbperfchk -i tcp::5000 -o null

will write to a file in current working directory

Or

nbperfchk -i tcp::5000 -o /dev/null

will discard to /dev/null

Verify using netstat that it is listening on the specified port before proceeding
 
TCP 0.0.0.0:5000 0.0.0.0:0 LISTENING 7276 [nbperfchk.exe]

Then start the writer that will connect to the reader. This will be from the source of the data.
nbperfchk -i zero: -o tcp:<reader server name>:5000

If you get the below output, then it is because it is blocked by the firewall, the read is not listening on the port used or the read process has terminated.
 
C:\BEPERFCHECK>nbperfchk -i zero: -o tcp:<servername>:5000 -s 32g

output: An operation was attempted on something that is not a socket.

Check the firewall settings or turn it off for the duration of the test and start over with the reader, verify that it is listening, and then proceed with the writer again.

In the example below a 32GB file is used which in most cases will be sufficient. This will create a null file of the size specified in the same folder path where the NBPerfcheck.exe is on the reader! You can cancel anytime with crtl+c, and if done on the writer it will also end the reader.
C:\BEPERFCHECK>nbperfchk -i zero: -o tcp:<servername>:5000 -s 32g
210 MB @ 70.1 MB/sec, 210 MB @ 70.1 MB/sec
527 MB @ 88.0 MB/sec, 317 MB @ 105.8 MB/sec
845 MB @ 94.0 MB/sec, 317 MB @ 106.0 MB/sec
1183 MB @ 98.6 MB/sec, 337 MB @ 112.5 MB/sec
|
|
32539 MB @ 112.2 MB/sec, 339 MB @ 113.1 MB/sec
32768 MB @ 111.8 MB/sec, 228 MB @ 76.2 MB/sec
32768 MB @ 112.2 MB/sec

In this example the network performance was 112 MB/sec. This is good performance.

Both the writer and reader will have output but the one that is interesting for this test is the reader output.
If this output is comparable to what is shown in the Backup Exec job log then the performance issue is not with the product used but the environment it is used in. Keep in mind technology overhead for GRT enable backups!
This test has also very good usage in conjunction with testing if RSS, NetDMA or TCP connection offload affects the performance (MS-KB2643970). Make one change and then retest. If enabling/disabling has no effect leave the settings at their default. Always make sure to be using the very latest NIC drivers before testing this!

When all the testing is done make sure to delete all the test files created by this tool!

 

 

If nbperfchk is downloaded from this article to run on client it can fail to run and errors with:

The code execution cannot proceed because pthreadVC3.dll was not found. Reinstalling the program may fix this problem.

You will have to download the pthreadVC3.dll from a Windows NetBackup server (master or media) from <install_path>\veritas\netbackup\bin and place it in the C:\windows\system32 directory of the client you are running nbperfchk from.

 

Install Path
- nbperfchk is already located in <install_path>\veritas\pdde\ on Windows and /usr/openv/pdde/pdag/bin on Linux on NetBackup Servers from version 10.x onward till 10.4  and Prior to this it is located at <install_path>\veritas\netbackup\bin\support and /usr/openv/netbackup/bin/support. From 10.5 onward, it is located at <install_path>\veritas\netbackup\bin\support on Windows and /usr/openv/netbackup/bin/support on Linux

- Starting 10.5, nbperfchk is supported on additional platforms as mentioned below-AMD64, linuxR_x86_4.18.0, linuxS_x86_4.4.73, plinuxR_4.18.0, plinuxS_5.3.18, rs6000_71, solaris10, solaris_x86_10_64, zlinuxR_4.18.0, zlinuxS_5.3.18.