Access 3340 Appliance - S3 software test fails

Problem

Access Appliance "Support> Test Hardware" fails to validate S3 service, or shows "Warning" on output

Error Message

"Test Software" may show errors messages like:

- [Warning] V-493-830-5451: The S3 service is not ready. Refer to the Access appliance documentation for how to configure the S3 service.

or

Appliance self-test result: FAIL

ACCESS SELFTEST ERROR V-493-10-0 Unable to create the ObjectAccess bucket. Contact Veritas Technical Support.

Example:

3340_01.Support> Test Software

Starting self-test at Mon Apr 22 08:07:46 2019

Running validation tests on the host '3340_01'.

...
Testing the S3 service ... [WARNING]

Testing the Access services ... [PASS]

...

Validation tests are complete.

Appliance self-test result: PASS

- [Warning] V-493-830-5451: The S3 service is not ready. Refer to the Access appliance documentation for how to configure the S3 service.

Cause

The Access S3 selftest does the following items via an API to test S3 functionality:

1. Get the name of the configured S3 server

2. Create a S3 secret key

3. Make a S3 bucket

4. Do PUT and GET operations on the S3 bucket

5. Delete the newly-created S3 bucket

6. Delete the newly created S3 key

The parameters for the operation to create the bucket is defined by the default bucket size, shown in the output of the Access CLISH > objectaccess show:

3340cluster> objectaccess show
Name           Value
=============  ===========================
Server Status  Enabled
Admin_URL      http://admin.3340cluster:8144
S3_URL         http://s3.3340cluster:8143
admin_port     8144
s3_port        8143
ssl            no
pools          naspool
fs_size        10g
fs_type        largefs-simple
fs_blksize     8192
fs_pdirenable  yes

When the API is called to create the bucket, it will attempt to make a bucket of the default size. 

NOTE: The default size is of the S3 bucket is set when creating a bucket using the "Provision Storage" wizard in the GUI.

If there is not spare space on the Access Appliance to create that bucket, this operation will fail. 

Example scenario:

A user has a single Access 3340 Appliance with ~630TB of space. 

They use the Storage Provisioning wizard to configure a bucket for S3 storage of 400TB. 

They then run the "Support> Test Software" to check on the health of the appliance, but the S3 test fails.  The test would fail because the API is attempting to make a new bucket of 400TB (which is now the default size), but there is only 230TB available.

Solution

Review and validate errors in storage_fs_create.log:

/opt/VRTSnas/log/storage_fs_create.log

Snippet from log:

vxassist -g sfsdg -oallowspansites make S3fs1555946361_tier1 389120G layout=concat site:naspool encrypted=off
writetolog ERROR 'Unable to create fs S3fs1555946361 due to either insufficient space/unavailable disk. Please run scanbus.'

From the CLISH and the above command, we can see , we can see that the "site" parameter is specifying the pool name.

3340cluster> storage pool free
Pool     Free Space  Total Space  Use%
=======  ==========  ===========  ====
naspool  645G        635.05T      99%

Only one device available with free space, 645GB free - not enough to create 389120G volume.
Setting default FS size to 10GB fixed issue.

You can set the default FS size in the GUI or Access management CLISH. 

CLISH command ex:

3340cluster> objectaccess set fs_size 10g

NOTE: This behavior will be altered in Access patch 7.4.2.2 (release date TBD) - S3 test will be skipped if bucket already exists.