Backup jobs are failing intermittently with error: premature eof encountered (233)

Problem

• Random backups will fail with error 233 or 40.
• Sometimes, the backup may be successful on the second attempt.
• This can impact different types of backups, such as VMware, SQL, Oracle, filesystem.
• The behavior includes failures at an individual media server or all media servers.

Error Message

The bpbrm process on the media server will send communication to the bpdbm process on the master server but it will fail with premature eof encountered (233) error:

00:04:12.562 [20228.8140] <2> logconnections: BPDBM CONNECT FROM XX.XXX.XX.XXX.50483 TO XX.XX.XX.XXX.1556 fd = 1020
00:04:12.562 [20228.8140] <2> db_begin: auth only query 78, socket 1020 is not proxied
00:04:12.562 [20228.8140] <2> get_exactly_n_bytes_or_eof_abs: premature EOF found
00:04:12.562 [20228.8140] <2> ts_get_long_abs: error reading long from socket: The process cannot access the file because it is being used by another process.  (32)
00:04:12.562 [20228.8140] <2> ts_get_adaptable_string: error reading long from socket: The process cannot access the file because it is being used by another process.  (32)
00:04:12.562 [20228.8140] <2> db_getdata: ts_get_string_handle() failed: The process cannot access the file because it is being used by another process.  (32)
premature end of file encountered (-9) WSAGetLastError(): 0
00:04:12.562 [20228.8140] <2> db_startrequest: protocol error 1
00:04:12.562 [20228.8140] <16> db_begin: db_startrequest() failed: premature eof encountered
00:04:12.562 [20228.8140] <2> db_FLISTsend: db_begin() failed: premature eof encountered
00:04:12.562 [20228.8140] <16> non_mpx_backup_archive_verify_import: db_FLISTsend failed: premature eof encountered (233)

The bpbrm log location at media server

• Unix: /usr/openv/netbackup/logs/bpbrm
• Windows: install_path\NetBackup\logs\bpbrm

The bpdbm process failed to receive the socket from vnet proxy:

00:04:12.719 [17856.24872] <2> DbmParentInChild::handle_input: Successfully received fd(1012) from parent(BPDBM ACCEPT FROM XX.XXX.XX.XXX.50483 TO
XX.XXX.XX.XXX.1556 fd = 908), request count(5)
00:04:12.719 [17856.24872] <2> daemon_proxy_proto: Preparing to do daemon protocol for (XX.XXX.XX.XXX:1556 <- XX.XXX.XX.XXX:50483)
00:04:12.734 [17856.24872] <16> vnet_proxy_handoff_receive: WSASocket() failed: 10107 <----------------------------------------
00:04:12.734 [17856.24872] <16> vnet_proxy_protocol_from_legacy_ex: vnet_proxy_handoff_receive() failed: 9
00:04:12.734 [17856.24872] <16> vnet_proxy_socket_swap: vnet_proxy_protocol_from_legacy() failed: 9
00:04:12.734 [17856.24872] <16> daemon_proxy_proto: vnet_proxy_socket_swap() failed: vnet status 9, nb status 23
00:04:12.734 [17856.24872] <16> process_request: Secure proxy protocol failed

The bpdbm log location at master server:

• Unix: /usr/openv/netbackup/logs/bpdbm
• Windows: install_path\NetBackup\logs\bpdbm

Cause

• The bpdbm process fails to receive the socket connection from the vnet proxy, which hints at an SSL related issue.
• A non-empty OpenSSL error queue may cause subsequent I/O failures and result in this error.
• Clearing the OpenSSL error queue before SSL_shutdown() in the vnetd proxy allows backups to complete.
• Since the communication between bpbrm and bpdbm is working sometimes, it is not related to certificates.

Solution

1. Ensure that steps from this article has been implemented at respective server(mention in the article itself).

   • Backups which contain a large number of files might intermittently fail with the Error db_FLISTsend failed: premature eof encountered (233)

2. Ensure that step 1 and step 2 from this article has been implemented at respective server.(mention in the article itself).

   • The nbpxyhelper log shows "(database is locked)"

3. Ensure that general recommendations for virus scanner exclusions are in place at the master and media server.

   • General recommendations for virus scanner exclusions working with NetBackup 

   • Note: If possible, disable or remove the antivirus from master and media server.

4. If the issue is still happening, an EEB exists which will help to clear the openssl error queue before SSL_shutdown().

   • The binary exists for the following versions of NetBackup:

     • Netbackup 8.1.1: Etrack 4017171
     • NetBackup 8.1.2: Etrack 4004942
     • NetBackup 8.2: Etrack 4007318

   • The EEB needs to be installed on the master server and all affected media servers.

Please contact Veritas Technical Support to obtain this fix. Please note that Veritas Technologies LLC reserves the right to remove any fix from the targeted release if it does not pass quality assurance tests. Veritas’ plans are subject to change and any action taken by you based on the above information or your reliance upon the above information is made at your own risk.

Please contact your Veritas Sales representative or the Veritas Sales group for upgrade information including upgrade eligibility to the release containing the resolution for this issue.