vxiod processes consuming high CPU, causing the server to hang intermittently 

Article: 100075427
Last Published: 2025-09-18
Ratings: 0 0
Product(s): InfoScale & Storage Foundation

Problem

vxiod processes consuming high CPU and causing the server to hang intermittently 

Error Message

vxiod shows below stack.

#3 [] end_repeat_nmi at
[exception RIP: native_queued_spin_lock_slowpath+290]
#4 [] native_queued_spin_lock_slowpath at
#5 [] queued_spin_lock_slowpath at
#6 [] _raw_spin_lock_irqsave at
#7 [] volrp_sendsio_start at

Example:

PID: 4909     TASK: ffff907ac2920000  CPU: 6    COMMAND: "vxiod"
 #0 [fffffe78f086ee60] crash_nmi_callback at ffffffff9ee734c1
 #1 [fffffe78f086ee68] nmi_handle at ffffffff9ee36fbb
 #2 [fffffe78f086eeb0] default_do_nmi at ffffffff9fb22560
 #3 [fffffe78f086eed0] exc_nmi at ffffffff9fb22760
 #4 [fffffe78f086eef0] end_repeat_nmi at ffffffff9fc0163c
    [exception RIP: native_queued_spin_lock_slowpath+50]
    RIP: ffffffff9fb37b12  RSP: ffffaa59c2063df0  RFLAGS: 00000002
    RAX: 0000000000000001  RBX: ffff907f466e8000  RCX: 0000000000000000
    RDX: 0000000000000001  RSI: 0000000000000001  RDI: ffff907cc1c84838
    RBP: ffff907cc1c84838   R8: ffff907ca2f29c00   R9: 000000008020001a
    R10: 0000000040000000  R11: ffff907fff82e600  R12: 0000000000000246
    R13: ffff907cc1c84838  R14: 0000000000000000  R15: ffff907acab80000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
--- <NMI exception stack> ---
 #5 [ffffaa59c2063df0] native_queued_spin_lock_slowpath at ffffffff9fb37b12                <<<< spin lock here.
 #6 [ffffaa59c2063e10] _raw_spin_lock_irqsave at ffffffff9fb372e0
 #7 [ffffaa59c2063e20] vol_rv_send_ack_start at ffffffffc1822690 [vxio]
 #8 [ffffaa59c2063ea8] voliod_iohandle at ffffffffc168e2cd [vxio]
 #9 [ffffaa59c2063ee8] voliod_loop at ffffffffc168e5dd [vxio]
#10 [ffffaa59c2063f18] kthread at ffffffff9ef3eaad
#11 [ffffaa59c2063f50] ret_from_fork at ffffffff9ee08549

 

Cause

Due to spinlock contention, vxiod processes with a similar stack are consuming high CPU when 64+ VVR connections are configured. This leads to CPU starvation, causing the server to intermittently enter a hang state.

Solution

The Product Engineering team currently plans to address this issue through a patch or hotfix in the current software version. Please note that our company reserves the right to withdraw any fix from the targeted release if it fails quality assurance tests. Development plans are subject to change, and any actions you take based on this information, or your reliance on it, are at your own risk.

The code change has been modified to reduce the spinlock contention included in VRTSvxvm 8.0.2.2607 HF. This fix should be installed on top of either the refreshed or non-refreshed InfoScale 8.0.2u6 patch (ie InfoScale 8.0.2.3200 )

Public links to access this patch: 

RHEL 8   - https://www.veritas.com/support/en_US/downloads/update.UPD951237 
RHEL 9   - https://www.veritas.com/support/en_US/downloads/update.UPD213702 
SLES 15 - https://www.veritas.com/support/en_US/downloads/update.UPD717762 

Note:

  • Arctera Infoscale Version 8.0.2U6 (build 8.0.2.3200) was refreshed on 13th September 2025 to incorporate critical VxFS updates, including VRTSvxfs-8.0.2.2800 and VRTSodm-8.0.2.2800. Please use the updated build for upgrade.
  • The patch identified as VRTSvxvm 8.0.2.2607 has some additional bug fixes. Refer to its README for more information.

 

Was this content helpful?