How to configure EKMS (External KMS) for MSDP storage

Description

How to configure EKMS (External KMS) for MSDP storage

Workflow to configure External KMS:-

Step 1 Validate KMS credentials & KMS compatibility

Step 2 Configure KMS credentials

Step 3 Configure KMS 

Step 4 Create keys

Step 5 Configure storage

Step 6 Configure policy

Workflow for external KMS configuration

Below are the detailed steps in the workflow:-

A) On Primary server

Go to /usr/openv/netbackup/bin:-

1) Validate KMS credentials and KMS compatibility:-

Validate Credentials:-

Run the following command:

nbkmiputil -validate -kmsServer <external_kms_server_name> -port port -certPath cert_path -privateKeyPath private_key_path -trustStorePath trust_store_path

To check if the NetBackup primary server is compatible with the KMS vendor and it can communicate with the KMS vendor using the KMIP protocol. Run the following command:

nbkmiputil -kmsServer <external_kms_server_name> -port port -certPath cert_path -privateKeyPath private_key_path -truststorepath trust_store_path -ekmsCheckCompat

2) Configure KMS Credentials:- 

./nbkmscmd -configureCredential -credName <credentialname> -privateKeyPath <privateKeyPath/key.pem> -certPath <certPath/cert_chain.pem> -trustStorePath <trustStorePath/cacerts.pem> -crlchecklevel <LEAF | CHAIN | DISABLE>

eg:-

./nbkmscmd -configureCredential -credName ekmsCred -privateKeyPath /usr/openv/var/gauravp/key.pem -certPath /usr/openv/var/gauravp/cert_chain.pem -trustStorePath /usr/openv/var/gauravp/cacerts.pem -crlchecklevel LEAF
Performing KMS Configuration health check...

Certificate Details
-------------------

Certificate #0:
Subject: CN=gauravp,O=Veritas,OU=extkms01
Issuer: CN=interCAkms,O=Veritas,OU=extkms01
Valid from: Mar 21 06:54:21 2021 GMT until Mar 26 06:54:21 2022 GMT

Certificate #1:
Subject: CN=interCAkms,O=Veritas,OU=extkms01
Issuer: CN=gemekms1,O=Veritas,OU=extkms01
Valid from: Feb 10 13:48:30 2021 GMT until Feb 15 13:48:30 2022 GMT

Validation Result
---------- ------
USER_INPUT_CERT_PATH_VALIDATION PASS
(Ensuring that the certificate path is valid)
USER_INPUT_TRUST_STORE_PATH_VALIDATION PASS
(Ensuring that the trust store path is valid)
USER_INPUT_PRIVATE_KEY_PATH_VALIDATION PASS
(Ensuring that the private key path is valid)
USER_INPUT_CERTIFICATES_VALIDATION PASS
(Ensuring that the certificates in certificate chain are readable)
USER_INPUT_TRUST_STORE_VALIDATION PASS
(Ensuring that the certificates in trust store are readable)
USER_INPUT_PRIVATE_KEY_READ_VALIDATION PASS
(Ensuring that the private key is readable)
CERTIFICATE_CN_EMPTINESS_VALIDATION PASS
(Ensuring that the Common Name field is not empty)
CERTIFICATE_CHAIN_EXPIRY_VALIDATION PASS
(Ensuring that the certificate is not expired)
CERTIFICATE_CHAIN_CURRENT_ACTIVE_VALIDATION PASS
(Ensuring that the certificate is currently valid)
PRIVATE_KEY_VALIDATION PASS
(Ensuring that the private key matches the certificate)
CERTIFICATES_ORDER PASS
(Ensuring that the certificates are in the appropriate order)
CRL_CDP_URL_VALIDATION PASS
(Ensuring that the CRL distribution point specifies HTTP / HTTPS URL)
Credential with ID 3de8379c-4a15-46ad-bd65-ea211f3bb63a is successfully created.

To list all credential details:-

nbkmscmd -listCredential

2) Perform pre-check of the EKMS configuration:- 

./nbkmscmd -precheckKMSConfig -privateKeyPath <privateKeyPath/key.pem> -certPath <certPath/cert_chain.pem> -trustStorePath <trustStorePath/cacerts.pem> -port 5696 -kmsServerName <external KMS server name>

eg:-

./nbkmscmd -precheckKMSConfig -privateKeyPath /usr/openv/var/gauravp/key.pem -certPath /usr/openv/var/gauravp/cert_chain.pem -trustStorePath /usr/openv/var/gauravp/cacerts.pem -port 5696 -kmsServerName virat.com
Certificate health check status:

Validation Result
---------- ------
USER_INPUT_CERTIFICATES_VALIDATION PASS
(The certificate(s) in certificate chain are readable.)
USER_INPUT_TRUST_STORE_VALIDATION PASS
(The certificate(s) in trust store are readable.)
USER_INPUT_PRIVATE_KEY_READ_VALIDATION PASS
(The private key is readable.)
CERTIFICATE_CN_EMPTINESS_VALIDATION PASS
(The Common Name field is not empty.)
CERTIFICATE_CHAIN_EXPIRY_VALIDATION PASS
(The certificate is not expired.)
CERTIFICATE_CHAIN_CURRENT_ACTIVE_VALIDATION PASS
(The certificate is currently valid.)
PRIVATE_KEY_VALIDATION PASS
(The private key matches the certificate.)
CERTIFICATES_ORDER PASS
(The certificates are in the appropriate order.)
CRL_CDP_URL_VALIDATION PASS
(The CRL distribution point specifies HTTP / HTTPS URL.)

Key management service status:
KMS Type - KMIP
KMS Server Name - virat.com
KMS Port - 5696
Connection status - 0 : the requested operation was successfully completed
Key listing status - 0 : the requested operation was successfully completed
Key Get status - 0 : the requested operation was successfully completed
Key get attribute status - 0 : the requested operation was successfully completed
Key set attribute status - 0 : the requested operation was successfully completed

Number of keys identified for NetBackup use = 12

The precheck operation to validate KMS configuration is successful.

3) Configure KMS:- 

./nbkmscmd -configurekms -name <configuration name for EKMS> -type kmip -port 5696 -credId <generated in step 1> -kmsServerName <external KMS server name> -priority 20

Or

./nbkmscmd -configurekms -name <configuration name for EKMS> -type kmip -port 5696 -credName <generated in step 1> -kmsServerName <external KMS server name> -priority 20

eg:-

./nbkmscmd -configurekms -name Flexscale_Ekms -type kmip -port 5696 -credId 3de8379c-4a15-46ad-bd65-ea211f3bb63a -kmsServerName virat.com -priority 20
The KMS configuration is successfully added in the NetBackup database.

4) Create a key & key group:- 

./nbkmscmd -createkey -name <configuration name for EKMS given in step 3> -keyName <new key name> -keyGroupName <new key group name>

Note:- keyName & keyGroup:- Whatever name you mention for the keyName and keyGroup in this command, it will be created on the external KMS tool/console automatically. 

Don't use keyGroup name as <stroage server:msdp> instead use "<storage server name-msdp>"

eg:-

./nbkmscmd -createkey -name Flexscale_Ekms -keyName ekmskey_v20 -keyGroupName c240m5-08-v20_kms_kg
The key with keyid [f68153ce5a2d469c9ce5729b73262408cfa514f5a9d547fc869a05775075cab8] is successfully created in the specified KMS.

5) Validate KMS configuration:-

./nbkmscmd -validateKMSConfig -name <configuration name for EKMS given in step 3>

eg:-

./nbkmscmd -validateKMSConfig -name Flexscale_Ekms
Certificate health check status:

Validation Result
---------- ------
USER_INPUT_CERTIFICATES_VALIDATION PASS
(The certificate(s) in certificate chain are readable.)
USER_INPUT_TRUST_STORE_VALIDATION PASS
(The certificate(s) in trust store are readable.)
USER_INPUT_PRIVATE_KEY_READ_VALIDATION PASS
(The private key is readable.)
CERTIFICATE_CN_EMPTINESS_VALIDATION PASS
(The Common Name field is not empty.)
CERTIFICATE_CHAIN_EXPIRY_VALIDATION PASS
(The certificate is not expired.)
CERTIFICATE_CHAIN_CURRENT_ACTIVE_VALIDATION PASS
(The certificate is currently valid.)
PRIVATE_KEY_VALIDATION PASS
(The private key matches the certificate.)
CERTIFICATES_ORDER PASS
(The certificates are in the appropriate order.)
CRL_CDP_URL_VALIDATION PASS
(The CRL distribution point specifies HTTP / HTTPS URL.)

Key management service status:
Configuration Name - GauravKms
KMS Type - KMIP
KMS Server Name - virat.com
KMS Port - 5696
Connection status - 0 : the requested operation was successfully completed
Key listing status - 0 : the requested operation was successfully completed
Key Get status - 0 : the requested operation was successfully completed
Key get attribute status - 0 : the requested operation was successfully completed
Key set attribute status - 0 : the requested operation was successfully completed

Number of keys identified for NetBackup use = 12

The specified KMS configuration is successfully validated.

B) On the Media server which is associated with the MSDP storage server:-

1) Configure storage properties related to external KMS:-

Create a file config_ekms on path /usr/openv & add the following entries:- 

V7.5 "operation" "set-local-lsu-kms-property" string
V7.5 "encryption" "1" string
V7.5 "kmsenabled" "1" string'
V7.5 "kmsservertype" "0" string
V7.5 "kmsservername" "Should be primary server name" string
V7.5 "keygroupname" "Created in step 4 of Primary server" string

Note : For Windows , config_ekms file entries should be: 

V7.5 "operation" "set-local-lsu-kms-property" string
V7.5 "encryption" "1" int
V7.5 "kmsenabled" "1" int
V7.5 "kmsservertype" "0" int
V7.5 "kmsservername" "Should be primary server name" string
V7.5 "keygroupname" "Created in step 4 of Primary server" string

Note:- Don't change any setting other than kmsservername & kmsgroupname. The kmsservername will be the "Primary server" only, it won't be external KMS server name. File path can be anything on the media server. 

Enabling KMS encryption for local LSU

2) Update storage properties with details as mentioned in step 1 of Media server:-

Go to /usr/openv/netbackup/bin/admincmd/ or <install path>\NetBackup\bin\admincmd & run the following command:-

./nbdevconfig -setconfig -storage_server <storage server name> -stype PureDisk -configlist /usr/openv/config_ekms
Storage server <storage server name> has been successfully updated

eg:-

./nbdevconfig -setconfig -storage_server c240m5-08-v20.com -stype PureDisk -configlist /usr/openv/config_ekms
Storage server c240m5-08-v20.com  has been successfully updated

3) Validate if the external KMS configuration is updated by running the below command:-

./nbdevconfig -getconfig -storage_server <storage server name> -stype PureDisk -l

It should show the following entries as mentioned in step 1 in configuration:-

V7.5 "encryption" "1" string
V7.5 "kmsenabled" "1" string'
V7.5 "kmsservertype" "0" string
V7.5 "kmsservername" "Should be primary server name" string
V7.5 "keygroupname" "Created in step 4 of Primary server" string

************* THE EKMS HAS BEEN CONFIGURED SUCCESSFULLY. ************

C) To cross check go to the following location:

1) Login to flex node

nodeb:/home/maintenance # docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bc752a05f555 uss-engine "init 'systemd.set..." 22 hours ago Up 22 hours (healthy) 10.xxxxxxx
5acfd7c16bd5 netbackup/main:latest "/usr/sbin/init" 22 hours ago Up 22 hours (healthy) nb_media
1e53cb82bb39 netbackup/main:latest "/usr/sbin/init" 23 hours ago Up 23 hours (healthy) nb_master
f337e47cddb7 uss-controller "/entrypoint.sh /u..." 23 hours ago Up 23 hours (healthy) uss-controller
7bb67a0c436e uss-proxy "/entrypoint.sh /u..." 23 hours ago Up 23 hours (healthy) uss-proxy
d95bdf9fe41b uss-mds "/uss-mds/uss_mds...." 23 hours ago Up 23 hours (healthy) cluster_mds_node
897a3f9bd720 asc-apiserver:latest "/usr/bin/java -Xm..." 23 hours ago Up 23 hours 0.0.0.0:18446->8446/tcp asc-apiserver
d63d8601ffd3 mongodbimage:latest "sh /usr/local/bin..." 23 hours ago Up 23 hours 27017/tcp asc-mongodb
79bfc2e26dbe rabbitmqimage:latest "sh /usr/local/bin..." 23 hours ago Up 23 hours 4369/tcp, 5671-5672/tcp, 25672/tcp asc-rabbitmq

2) Check the storage server container id & login to bash as below

nodeb:/home/maintenance # docker exec -it 10.xxxxxx bash

3) Check KMS status, getmode, keydictutil:-

engine :

1) [msdp-15.0.2] r7515-093-vm06 >  setting encryption kms-status

INFO: KMS enabled

Operation completed successfully

2) engine : /usr/openv/pdde/pdcr/bin/crcontrol --getmode
Mode : GET=Yes PUT=Yes DEREF=Yes SYSTEM=Yes STORAGED=Yes REROUTE=No COMPACTD=Yes FIPS=Yes KMS=Yes

3) engine : /usr/openv/pdde/pdcr/bin/keydictutil --list

dsid: 0 encryption: Yes kms: Yes

key group name: <Should ekms key group name which was created in step 4 of primary server>
kms server name: <Should show primary server name which mentioned in step 1 of the media server>
readonly: No

4) engine : cd /msdp/data/dp1/pdvol/log/spoold
engine : cat spoold.log | grep -i lazy
June 01 20:38:47 INFO [140060868323072]: LazyInit: KMS is not initialized for MSDP, will start initializing KMS for MSDP
June 01 20:38:47 INFO [140060868323072]: LazyInit,the original max open file num soft is 1048576, hard is 1048576
June 01 20:38:47 INFO [140060868323072]: LazyInit: Start initializing KMS for MSDP
June 01 20:38:47 INFO [140060868323072]: LazyInit: Succeed to Load keydict for MSDP
June 01 20:38:50 INFO [140060868323072]: LazyInit: Succeed to initialize KMS for MSDP
June 01 20:38:50 INFO [140060868323072]: LazyInit, reset the max open file num soft is 1048576, hard is 1048576
June 01 20:38:50 INFO [140060868323072]: LazyInit, verify set max open file num soft is 1048576, hard is 1048576
engine :
engine :

D) Test Backup

In detail status & bptm logs it will show you below line:-

StorageServer=PureDisk:xxxxxxxxxxxx; Report=PDDO Stats (multi-threaded stream used) for (xxxxxxxxxxxxxx:PureDiskVolume): scanned: 13005866 KB, CR sent: 5789138 KB, CR sent over FC: 0 KB, dedup: 55.5%, cache hits: 0 (0.0%), where dedup space saving:0.0%, compression space saving:55.5%, new transferred data encrypted

E) Commands for reference to check and\or update

1) To check the KMS configuration on the Primary server:-
Go to /usr/openv/netbackup/bin/

./nbkmscmd -listkmsconfig

2) To update EKMS for Enable for backup true/false
./nbkmscmd -updatekmsconfig -name <config name> -EnabledforBackup 0
The specified KMS configuration is successfully updated.

Specifies whether keys from this KMS should be used for backup or not. The default value is 1.
Provide 0 if the keys from this KMS should not be used for backup.

3) To list keys from NetBackup:-

nbkmscmd -listKeys -name <config name>

4) To list keys from External KMS server:-

nbkmiputil -getKey -kmsServer <external kms server name> -port <port> -privateKeyPath <privateKeyPath/key.pem> -certPath <certPath/cert_chain.pem> -trustStorePath <trustStorePath/cacerts.pem> -nbKeyGroup key_group_name

Note:- KeyGroup name can be seen in External KMS portal under Meta/MetaData tab of the particular key.

Or

nbkmiputil -getKey -kmsServer <external kms server name> -port <port> -privateKeyPath <privateKeyPath/key.pem> -certPath <certPath/cert_chain.pem> -trustStorePath <trustStorePath/cacerts.pem> -keyId <keyid>

5) To list keyId from external keys:-

nbkmiputil -listKeyIDs -kmsServer <external kms server name> -port <port> -privateKeyPath <privateKeyPath/key.pem> -certPath <certPath/cert_chain.pem> -trustStorePath <trustStorePath/cacerts.pem>

6) To inactivate an external key:-

nbkmiputil -setState -kmsServer <external kms server name> -port <port> -privateKeyPath <privateKeyPath/key.pem> -certPath <certPath/cert_chain.pem> -trustStorePath <trustStorePath/cacerts.pem> -keyId <keyid> -stateType 2 -revocationReason 1

Command Guide