How to understand Veritas File System (VxFS) limits

Article: 100047118
Last Published: 2020-07-09
Ratings: 9 1
Product(s): InfoScale & Storage Foundation

Description

 

A file system is a method for storing and organizing files and the data they contain to make it easy to find and access them.

The placement of the filesystem structures & organization of user data on disk is referred to as the disk layout. Different disk layout versions provide new features with each Veritas File System (VxFS) product release.


The VxFS file system is consists of 2 or more file sets (filesystem checkpoints)

STRUCTURAL file set carrying index 1 accesses the “objects” storing general file system metadata

UNAMED file set is visible by the virtual file system and in use by an application (given index 999)




The UNAMED file set is used to store the file objects used by the application:




A file system is a set of abstract data types (such as metadata) that are implemented for the storage, hierarchical organization, manipulation, navigation, access, and retrieval of data.



Even though a VxFS file system can support 1 billion files per file system, the limit is practically limited on inodes, which includes more than just the regular files. Things like directories and symlinks also need one inode each.

An inode describes file system object information about a regular file or a directory (also known as metadata). The metadata is an efficient way of storing data, so it can referenced later. Each inode is 256 Bytes in size.


Veritas File System (VxFS) allocates disk space to files in groups of one or more adjacent blocks called extents. In simple terms, an extent is a contiguous area of storage in a file system, reserved for a file. Veritas File System (VxFS) makes it possible to pre-allocate space to a file at the time of the request rather than when data is written into the file.

When starting to write to a file, a whole extent is allocated. When writing to the file again, the data continues where the previous write left off. This reduces or eliminates file fragmentation and potentially enhances file system throughput by allowing disk I/O to take place in units of multiple blocks (if contiguous).

This space cannot be allocated to other files in the file system. VxFS prevents any unexpected out-of-space condition on the file system by ensuring that a file's required space will be associated with the file before it is required.

Files with large extents tend to be more contiguous and have better I/O characteristics.

The block size can have major benefits and disadvantages


Small block size can result in the following operational impact

 

  • Fragmentation of files due to extent allocation throughout file system
  • Fragmentation of free space
  • Resulting in performance impact

 

The number of inodes is a major factor in the runtime of a full fsck.

Placing files in a single directory can impact performance:

 

  • Performance hit during file lookups
  • Risk of LDH (Large Directory Hash) corruption, panics
  • Recovery time impact

 

Hundreds or even thousands of very large, pre-allocated files with contiguous extents are much easier to handle than hundreds of millions of medium sized files.

The maximum size of the file system you can create depends on the block size and is different for each platform.


VxFS supports creating a 16PB file system on Linux with 8kB block size, however, due to practical reasons we would NOT normally recommend having a file system anywhere near this size.

On AIX and Solaris, we still enforce the old 256TB limit.

Even on Linux, Veritas would NOT recommend regular customers exceeding more than 256TB. 

 

Block Size                                                           Sector Size                                                         Max File System Size    

1024 Bytes          (1K)                                       68,719,472,624 sectors                                 32 Tb

2048 Bytes         (2k)                                        137,438,945,248 sectors                               64 Tb

4096 Bytes         (4k)                                        274,877,890,496 sectors                               128 Tb

8192 Bytes         (8k)                                        529,755,780,992 sectors                               256 Tb

 

The default block size is 1Kb for file system sizes of up to 2Tb, and 8Kb for file system sizes 2Tb or larger.

You specify the block size when creating a file system by using the mkfs-o bsize option. The block size cannot be altered after the file system is created.


Choose a block size based on the type of application being run.

For example, if there are many small files, a 1 KB block size may save space.
For large file systems, with relatively few files, a larger block size is more appropriate.

Larger block sizes use less disk space in file system overhead, but consume more space for files that are not a multiple of the block size. The easiest way to judge which block sizes provide the greatest system efficiency is to try representative system loads against various sizes and pick the fastest.

 

The latest supported details are:
The maximum VxFS file system size is 256TB.

However, for Linux starting from VxFS version 7.4.1 the maximum size per vxfs file system is 5PB.

 

Max number of files per vxfs file system is 1 billion

 

The entire file system space, starting at block 0, is divided into fixed size allocation units (AU) of 32,678 blocks (32k)

Max throughput is entirely based on the hardware and the environment

 


Recovery time


A major factor to consider the size of your file system, is the recovery of the file system in the event that an fsck or metasave needs to be actioned to repair a damaged file system.

 

The VxFS file system uses a full structural verification utility called fsck to recover a damaged file system following a system failure (unplanned outage). This involves a time-consuming process of checking the entire file system structure, verifying that the file system is intact, and correcting any inconsistencies. VxFS reduces recovery times by tracking file system activity in the VxFS intent log. This feature records pending changes to the file system structure in a circular intent log. The mount command automatically runs the VxFS fsck command to perform an intent log replay if the mount command detects a dirty log in the file system.

The VxFS intent log is allocated when the file system is first created and the size is determined by the size of the file system. The larger the file system, the larger the intent log.

 

For the number of files in a directory, certain operations on a directory are still linear, so it is far better to deploy configurations referencing multiple directories and smaller sized file systems. Anything that uses readdir still has to just scan the whole directory. Imagine constantly doing readdir on the one main incoming directory with huge numbers of files.

 

The inode allocator was NOT designed with large inode counts in mind and will have some performance issues. Storage checkpoints are a simple linear multiple on the number of inodes.

When a VxFS file system is created using the mkfs command, a special object is created named the superblock.

The superblock contains fundamental size information, a summary of available resources & references to other places on disk where additional structural information can be found


NOTE: The file system will not mount without a valid superblock

The first copy of VxFS superblock is always located at Byte offset 8192(8192B) regardless of the file system block size. This ensures the VxFS file system type can always locate the superblock when mounting (if not corrupt). A secondary copy of the superblock consists at an alternate (unfixed) location

 
Veritas recommends to limit the number of files in a single VxFS directory to 100,000 files.  

This will avoid directory performance issues, e.g. taking long time to list the files in the directory, taking long time to create new files and rename existing files.  

Limiting the number of files to the recommended value is especially important if the directory is shared through NFS.

To maximize VxFS performance, do not exceed 100,000 files in the same directory.
Use multiple directories instead.

 

 

Was this content helpful?