Impact of Apache Log4j Vulnerabilities on NetBackup Flex Scale

Impact of Apache Log4j Vulnerabilities on NetBackup Flex Scale

Article: 100052101
Last Published: 2022-01-04
Ratings: 0 0
Product(s): Appliances

 

Impact of CVE-2021-44228, CVE-2021-45046, CVE-2021-45105 Apache Log4j Vulnerability Mitigation for Flex Scale Appliance  

 

About CVE-2021-44228, CVE-2021-45046, CVE-2021-45105 Apache Log4j Vulnerabilities 

Apache Log4j is an open-source, Java-based logging utility widely used by enterprise applications and cloud services. 

The Apache Software Foundation has released a security advisory to address a remote code execution vulnerability (CVE-2021-44228) and a denial of service vulnerability (CVE-2021-45046) affecting Log4j versions 2.0-beta9 to 2.15. A remote attacker could exploit these vulnerabilities to take control of an affected system.   

More information is available from the Apache Announcement and recommends upgrading to the latest Log4j 2.16.0 or applying recommended mitigations immediately. 

Issue 

CVE-2021-44228: Apache Log4j2 JNDI features do not protect against attacker controlled LDAP and other JNDI related endpoints. 

Severity: Critical 
Base CVSS Score: 10.0 
CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H 

 

CVE-2021-45046: Apache Log4j2 JNDI features do not protect against malicious input data using a JNDI Lookup pattern resulting in a denial of service (DOS) attack.  

Severity: Critical 
Base CVSS Score: 9.0 

CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:C/C:H/I:H/A:H 

 

CVE-2021-45105: Apache Log4j2 does not always protect from infinite recursion in lookup evaluation.  Fixed in Log 4j 2.17.0. Apache recommends upgrading to Log4j 2.17.0. NetBackup Flex Scale is not impacted by this vulnerability 

 

CVE-2021-4104 

JMSAppender in Log4j 1.2 is vulnerable to deserialization of untrusted data when the attacker has write access to the Log4j configuration.  Fixed in Log 4j 2. Apache recommends upgrading to Log4j 2. NetBackup Flex Scale Appliances are not exploitable by this vulnerability. 

 

CVE-2021-44832 

Apache Log4j2 versions 2.0-beta7 through 2.17.0 (excluding security fix releases 2.3.2 and 2.12.4) are vulnerable to a remote code execution (RCE) attack where an attacker with permission to modify the logging configuration file can construct a malicious configuration using a JDBC Appender with a data source referencing a JNDI URI which can execute remote code.  Fixed in Log 4j 2.17.1. Apache recommends upgrading to Log4j 2.17.1. NetBackup Flex Scale Appliances are not exploitable by this vulnerability. 

 

Affected NetBackup Appliance Versions: 

Product / Component 

Version 

Mitigation Steps 

Download EEB from  

NetBackup Flex Scale 

1.3 or 1.3.1 

Veritas strongly recommends customers upgrade to NetBackup Flex Scale 2.1 release in order to be able to perform the mitigation steps below. 

UPD195687 

 

NetBackup Flex Scale  

2.1 

 

Set parameter log4j2.formatMsgNoLookups to true 

UPD195687 

  

 

Mitigation for Flex Scale 2.1 

Pre-requisite: You should have nbfs 2.1 GA iso deployed on all nodes of your cluster. After cluster configuration follow below mentioned steps to install the EEBs..  

 

1. Download the VRTSnbfsapp_EEB_ET4059862-2.1.0.0-1.x86_64.rpm eeb from UPD195687 

2. Login to Appliance Management GUI (https://<Management_Server_FQDN>:14161) 

3. Navigate to Settings → Software Management page 

4. Click on "Add-ons" tab  

5.  Click on "Add" link and then click "choose add-ons" to upload the downloaded EEB from your local machine 

6.  Once the EEB rpm package is uploaded successfully, it will list there in the GUI table on that page 

7. Click on the radio button corresponding to your EEB which you want to install, and then click on "Install" button 

8. It will start the EEB installation, keep checking the progress of install from Recent Activity → View All activities task table 

9. Once EEB installation completes, it will automatically start full discovery to sync all the tasks/processes 

10. Wait for full discovery to complete  

11. Elevate to root. If lockdown mode is configured, please contact Veritas Support to get access to root shell. 

12. Verify that log4j class is deleted from asc-apiserver and asc-globalapiservice containers using the below steps : 

  • Login into the node through ssh using an Appliance Administrator account 

  • Run “Support elevate” and elevate to root shell 

  • Run hacli -cmd "docker exec asc-globalapiservice grep -r JndiLookup.class /opt"  

  • Run hacli -cmd "docker exec asc-apiserver grep -r JndiLookup.class /opt" 

  • Ensure the output of above commands doesn’t show any references to the effected JndiLookup class 

  • Verify asc-apiserver and asc-globalapiservice is running properly ‘docker ps -a’. 

  • Verify one of the nodes has globalapiservice running.  You can check that from this command ‘hastatus -sum | grep GLOBAL_API_SERVER | grep ONLINE’. 

13. Above steps can also be run on newly added node or replaced node to verify EEB fixes are available on these nodes. 

 

Manual mitigation steps for Flex Scale 2.1, if unable to apply the EEB:  

  1. Please run all the steps on all the nodes in the cluster unless stated otherwise.  

  1. Elevate to root. If lockdown mode is configured, please contact Veritas Support to get access to root shell. 

  1. Note tag details for asc-apiserver and asc-globalapiservice using ‘docker images’ these two images will be available on all the nodes and one can note them down from any one node.  

  1. Note down Entrypoint’s value section from ‘docker inspect asc-apiserver’ and ‘docker inspect asc-globalapiservice’ [NOTE: asc-globalapiservice will only be running on one node and one can find that node using this command ‘hastatus -sum | grep GLOBAL_API_SERVER | grep ONLINE’] 

  1. Create a file /tmp/asc-apiserver/Dockerfile and navigate to /tmp/asc-apiserver. 
    mkdir -p /tmp/asc-apiserver 
    cd /tmp/asc-apiserver 
    vi Dockerfile 

  1. Paste below contents in this Docker file. Update the Entry point details with details which you note down in step 2 removing tabs and new line characters.  

FROM asc-apiserver:latest 

RUN /bin/bash -c 'for log4jcore in `find /opt -name \*log4j\*core\*.jar 2> /dev/null`;do echo $log4jcore; zip -q -d $log4jcore org/apache/logging/log4j/core/lookup/JndiLookup.class;done' 

ENTRYPOINT ["/usr/bin/java", "--module-path=/opt/autosupport/apiserver/lib/ccj-3.0.1.jar", "--add-modules=ccj", "--add-exports", "java.base/sun.security.provider=ccj", "--add-exports", "java.base/sun.security.internal.spec=ccj", "-Dcom.safelogic.cryptocomply.fips.approved_only=true", "-Xmx1024m", "-XX:MaxMetaspaceSize=256m", "-cp", "/opt/autosupport/apiserver/ApiServer-2.4.0.jar:/opt/autosupport/apiserver/lib/*", "com.veritas.appliance.autosupport.client.apiserver.ApiServerMain"] 

  1. Run below command with the noted tag number in step 1. In this case it would be 2.4.0. 
    “docker build -t asc-apiserver:<tag_number> .”   

  1. Run ‘systemctl restart autosupportclient’. 

  1. Create a file /tmp/asc-globalapiservice/Dockerfile and navigate to /tmp/asc- globalapiservice. 
    mkdir -p /tmp/asc-globalapiservice 
    cd /tmp/asc- globalapiservice 
    vi Dockerfile 

  1. Paste below contents in this Docker file. Update the Entry point details with details which you note down in step 2 removing tabs and new line characters.  

FROM asc-globalapiservice:latest 

RUN /bin/bash -c 'for log4jcore in `find /opt -name \*log4j\*core\*.jar 2> /dev/null`;do echo $log4jcore; zip -q -d $log4jcore org/apache/logging/log4j/core/lookup/JndiLookup.class;done' 

ENTRYPOINT ["./bin/globalapiservice"] 

  1. Run below command with the noted tag number in step 1. In this case it would be 2.4.0. 
    “docker build -t asc-globalapiservice:<tag_number> .”   

  1. . Only perform the below steps on the machine that has globalapiservice running. You can check that from this command ‘hastatus -sum | grep GLOBAL_API_SERVER | grep ONLINE’.   
    /opt/VRTS/bin/hagrp -offline GLOBAL_API_SERVER -sys <node-name> 
    /opt/VRTS/bin/hagrp -state GLOBAL_API_SERVER (Make sure all globalapiserver are offline) 
    /opt/VRTS/bin/hagrp -online GLOBAL_API_SERVER -sys <node-name> 

  1. Verify asc-apiserver and asc-globalapiservice is running properly ‘docker ps -a’. 

  1. Verify if test email is successful from webgui. 

  1. Also, verify if you can see hardware health information on webgui. 

  1. Steps need to be repeated on the new node which is added to the cluster using the NetBackup Flex Scale Web interface. 

  1. Similarly, steps also need to be repeated on the node which is replaced in the cluster using the NetBackup Flex Scale Web interface. 

Flex Scale primary server container instructions (updated 2021-12-13) 

NOTE: The steps outlined in this article will require connecting to the NetBackup Primary/Master Application server with a NetBackup administrator user, usually with `ssh`. For example: `ssh nbadmin@nbuprimary.example.com`

Mitigation for Flex Scale 2.1 

Steps for testing Primary server container Log4j Version: 
************************************************************ 
Pre-requisite : You should have nbfs 2.1 GA iso deployed on all nodes of your cluster. After cluster configuration follow below mentioned steps to install the eebs 

 

Mitigation using EEB for NetBackup Flex Scale 2.1 

1. Download the VRTSnbfsapp_nb_EEB_ET4058560-9.1.0.1-2.x86_64.rpm eeb from UPD195687 

2. Login to Appliance Management GUI (https://<Management_Server_FQDN>:14161) 

3. Navigate to Settings → Software Management page 

4. Click on "Add-ons" tab  

5.  Click on "Add" link and then click "choose add-ons" to upload the downloaded EEB from your local machine 

6.  Once the EEB rpm package is uploaded successfully, it will list there in the GUI table on that page 

7. Click on the radio button corresponding to your EEB which you want to install, and then click on "Install" button 

8. It will start the EEB installation, keep checking the progress of install from Recent Activity → View All activities task table 

9. Once EEB installation completes, it will automatically start full discovery to sync all the tasks/processes 

10. Wait for full discovery to complete  

 

11. Find the node where NBU primary server is running and then login Inside Netbackup Primary container on that node using below command : 
command : docker exec -it nb_master /bin/sh 

 
12. Once you are inside the container, find the version of log4j using below command : 
command : sudo find / -name 'log4j*' 

 
13. Output of above command should show the log4j related jar files version as 2.16 

 

Manual mitigation steps for Flex Scale 2.1, if unable to apply the EEB:  

Section 1 

STEPS IF ATTEMPTING REMEDIATION FOR THE FIRST TIME  

NOTE: There is example output in the steps below. Please note that your output may differ and that important information has been called out where applicable. 

Step 1: Add mitigation to `/etc/environment` file 

For NetBackup Flex Scale versions 2.1 

Append the mitigation environment variable to `/etc/environment` 

$ sudo bash -c 'sudo echo LOG4J_FORMAT_MSG_NO_LOOKUPS="true" >> /etc/environment' 

$ cat /etc/environment 

LOG4J_FORMAT_MSG_NO_LOOKUPS=true 

 

NOTE: There may be more entries in the `/etc/environment` file. 

 

Create an `environment` file in the persistent location so the mitigation will be in place after instance restart: 

 

$ sudo bash -c 'echo LOG4J_FORMAT_MSG_NO_LOOKUPS="true" > /mnt/nbdata/vxos/etc/environment' 

$ cat /mnt/nbdata/vxos/etc/environment 

LOG4J_FORMAT_MSG_NO_LOOKUPS=true 

 

Restart nbwmc to activate the mitigation: 

$ sudo /usr/openv/netbackup/bin/nbwmc stop 

$ sudo /usr/openv/netbackup/bin/nbwmc start 

 

Verification that the environment variable is in effect: 

Run bpps to find the PID of the NetBackup webservice application: 

$ sudo /usr/openv/netbackup/bin/bpps -a |grep java 
nbwebsvc 13138 1 64 22:41 ? 00:03:27 /usr/openv/java/jre/bin/java -Dnop -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dnbwmc -d64 -Xms4096m -Xmx4096m -XX:MetaspaceSize=192m -XX:NewRatio=1 -XX:SurvivorRatio=4 -XX:-UseAdaptiveSizePolicy -Djava.library.path=/usr/openv/lib -DNB_HOSTNAME=deecl01vm054p5.engba.veritas.com -DNB_INSTALL_DIR=/usr/openv -DVAR_GLOBAL=/usr/openv/var/global -Djava.util.logging.config.file=/usr/openv/wmc/config/tomcat.logging.properties -Dlog.conf.path=/usr/openv/netbackup/nblog.conf -Dvrtsat.home=/usr/openv/netbackup/sec/at -Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.rejectClientInitiatedRenegotiation=true -Dcom.safelogic.cryptocomply.fips.approved_only=false -verbose:jni -Djava.security.egd=file:/dev/./urandom -XX:ErrorFile=/usr/openv/wmc/webserver/logs/hs_err_pid%p.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/openv/wmc/webserver/logs -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -Xloggc:/usr/openv/wmc/webserver/logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=1M -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/openv/wmc/webserver/bin/bootstrap.jar:/usr/openv/wmc/webserver/bin/tomcat-juli.jar -Dcatalina.base=/usr/openv/var/global/wsl/webserver -Dcatalina.home=/usr/openv/wmc/webserver -Djava.io.tmpdir=/usr/openv/wmc/webserver/temp org.apache.catalina.startup.Bootstrap start 

 

Using PID 13138 from the example output, grep for the environment variable in the running process environ. 

$ sudo -u nbwebsvc cat /proc/13138/environ | strings | grep LOG4J_FORMAT_MSG_NO_LOOKUPS 
LOG4J_FORMAT_MSG_NO_LOOKUPS=true 

 

Step 2: Create mitigation for systemd service 

 

Create an override directory: 

 

$ sudo mkdir -pv /etc/systemd/system/netbackup.service.d/ 

 

Create an override file for the log4j2 mitigation: 

 

$ sudo bash -c 'cat <<END_CONTENT > /etc/systemd/system/netbackup.service.d/log4j2.conf 

[Service] 

Environment="LOG4J_FORMAT_MSG_NO_LOOKUPS=true" 

END_CONTENT 

NOTE: Please pay special attention to the quoting and the ending END_CONTENT on a line by itself. 

 

Verify file content: 

 

$ sudo cat /etc/systemd/system/netbackup.service.d/log4j2.conf 

[Service] 

Environment="LOG4J_FORMAT_MSG_NO_LOOKUPS=true" 

 

Check the netbackup service: 

 

$ systemctl status netbackup 

● netbackup.service - LSB: NetBackup 

   Loaded: loaded (/etc/rc.d/init.d/netbackup; static; vendor preset: disabled) 

  Drop-In: /etc/systemd/system/netbackup.service.d 

           └─orchestration.conf 

   Active: active (running) since Mon 2021-12-13 23:21:22 UTC; 5h 19min ago 

     Docs: man:systemd-sysv-generator(8) 

  Process: 7652 ExecStart=/etc/rc.d/init.d/netbackup start (code=exited, status=0/SUCCESS) 

   CGroup: /system.slice/docker-f4203999fe84af43b18be0bff7bb773e1923da1880628a56bb9cdf6ffbb54629.scope/system.slice/netbackup.service 

           ├─ 7660 /usr/openv/netbackup/bin/vnetd -standalone 

 

-->8 snip 8<-- 

 

           └─25306 /usr/openv/netbackup/bin/bpdbm 

Warning: netbackup.service changed on disk. Run 'systemctl daemon-reload' to reload units. 

 

We expect to see the "Warning: netbackup.service changed on disk. Run 'systemctl daemon-reload' to reload units." message indicating the new log4j2.conf has been noticed. 

 

Execute 'systemctl daemon-reload' as suggested: 

 

$ sudo systemctl daemon-reload 

 

Check the service again and the "Warning" should not be observed anymore. Additionally, the log4j2.conf file will be listed in the "Drop-In" section. 

 

$ systemctl status netbackup 

● netbackup.service - LSB: NetBackup 

   Loaded: loaded (/etc/rc.d/init.d/netbackup; static; vendor preset: disabled) 

  Drop-In: /etc/systemd/system/netbackup.service.d 

           └─log4j2.conf, orchestration.conf 

   Active: active (running) since Mon 2021-12-13 23:21:22 UTC; 5h 21min ago 

     Docs: man:systemd-sysv-generator(8) 

   CGroup: /system.slice/docker-f4203999fe84af43b18be0bff7bb773e1923da1880628a56bb9cdf6ffbb54629.scope/system.slice/netbackup.service 

           ├─ 7660 /usr/openv/netbackup/bin/vnetd -standalone 

 

-->8 snip 8<-- 

 

           └─25306 /usr/openv/netbackup/bin/bpdbm 

 

Finally, use `systemctl cat netbackup` to view the unit file contents: 

 

$ systemctl cat netbackup 

# /etc/systemd/system/netbackup.service 

# Automatically generated by systemd-sysv-generator 

# Copied here to update from 5min default timeout to 15min 

 

[Unit] 

Documentation=man:systemd-sysv-generator(8) 

SourcePath=/etc/rc.d/init.d/netbackup 

Description=LSB: NetBackup 

Before=shutdown.target 

After=network-online.target 

After=vxpbx_exchanged.service 

Wants=network-online.target 

Conflicts=shutdown.target 

 

[Service] 

Type=forking 

Restart=no 

TimeoutSec=15min 

IgnoreSIGPIPE=no 

KillMode=process 

GuessMainPID=no 

RemainAfterExit=yes 

ExecStart=/etc/rc.d/init.d/netbackup start 

ExecStop=/etc/rc.d/init.d/netbackup stop 

# /etc/systemd/system/netbackup.service.d/log4j2.conf 

[Service] 

Environment=”LOG4J_FORMAT_MSG_NO_LOOKUPS=true" 

 

Notice the new section with the log4j2.conf file: 

# /etc/systemd/system/netbackup.service.d/log4j2.conf 

[Service] 

Environment=”LOG4J_FORMAT_MSG_NO_LOOKUPS=true" 

 

Note: the file log4j2.conf in the location above WILL NOT persist across restarts. 

To persist the file, follow these steps: 

 

$ sudo mkdir -pv /mnt/nbdata/vxos/etc/systemd/system 

mkdir: created directory '/mnt/nbdata/vxos/etc/systemd' 

mkdir: created directory '/mnt/nbdata/vxos/etc/systemd/system' 

 

$ sudo cp -prv /etc/systemd/system/netbackup.service.d/ /mnt/nbdata/vxos/etc/systemd/system 

‘/etc/systemd/system/netbackup.service.d/’ -> ‘/mnt/nbdata/vxos/etc/systemd/system/netbackup.service.d’ 

‘/etc/systemd/system/netbackup.service.d/log4j2.conf’ -> ‘/mnt/nbdata/vxos/etc/systemd/system/netbackup.service.d/log4j2.conf’ 

 

With those steps completed, the override IS persisted.  

 

Section 2: To revert the environment variable changes, follow these instructions. 

 

Remove the 'LOG4J_FORMAT_MSG_NO_LOOKUPS="true"' entry from /etc/environment 

 

$ cat /etc/environment 

LOG4J_FORMAT_MSG_NO_LOOKUPS=true 

 

$ sudo sed -i.bak -e 's|LOG4J_FORMAT_MSG_NO_LOOKUPS=true||g' /etc/environment 

 

$ cat /etc/environment 

 

Remove the /mnt/nbdata/vxos/etc/environment file 

 

$ sudo rm -v /mnt/nbdata/vxos/etc/environment 

removed ‘/mnt/nbdata/vxos/etc/environment’ 

 

Remove the systemd service override 

 

$ ls -l {'',/mnt/nbdata/vxos}/etc/systemd/system/netbackup.service.d/ 

/etc/systemd/system/netbackup.service.d/: 

total 8 

-rw-r--r--. 1 root root 57 Dec 14 04:37 log4j2.conf 

-rw-r--r--. 1 root root 50 Dec 13 23:15 orchestration.conf 

 

/mnt/nbdata/vxos/etc/systemd/system/netbackup.service.d/: 

total 16 

-rw-r--r--. 1 root root 57 Dec 14 04:37 log4j2.conf 

-rw-r--r--. 1 root root 50 Dec 13 23:15 orchestration.conf 

 

$ sudo rm -v /etc/systemd/system/netbackup.service.d/log4j2.conf 

removed ‘/etc/systemd/system/netbackup.service.d/log4j2.conf’ 

 

Remove the file from persistent storage: 

 

$ sudo rm -v /mnt/nbdata/vxos/etc/systemd/system/netbackup.service.d/log4j2.conf 

 

Reload the unit files: 

 

$ sudo systemctl daemon-reload 

 

If desired, stop/start nbwmc: 

 

$ sudo /usr/openv/netbackup/bin/nbwmc stop 

$ sudo /usr/openv/netbackup/bin/nbwmc start 

 

Please note that permanent fix for CVE-2021-44228 and CVE-2021-45046 is on the way. 

 

 

 

Was this content helpful?