Troubleshooting Internal Server Error Caused by Storage Layout

Introduction

One of the more frustrating issues I encountered while deploying Security Analytics on custom hardware was a recurring Internal Server Error within the web interface.

Initially, the appliance functioned normally. However, periods of increased traffic would eventually cause the /var filesystem to exceed 80% utilization before ultimately filling to 100%, at which point the web interface would begin returning Internal Server Error messages.

Broadcom has published several Knowledge Base articles covering common causes of this problem.

Error “Internal Server Error” on Security Analytics login page or Summary widgets
- https://knowledge.broadcom.com/external/article/168502/error-internal-server-error-on-security.html
/var is full due to large audit.log files
- https://knowledge.broadcom.com/external/article/168496
/var partition is filling up or is 100% utilized
- https://knowledge.broadcom.com/external/article/168963/var-partition-is-filling-up-or-is-100-ut.html

While those articles are useful, none of them resolved the issue in our environment.

After comparing our appliance against factory-built Security Analytics hardware provided by Broadcom, we discovered that the storage layout created during installation was missing several expected partitions and logical volumes.

This guide documents how we corrected the storage layout and eliminated the recurring Internal Server Error.

Warning

This procedure is destructive. It removes partitions, volume groups, and logical volumes from the storage device being modified. Ensure all required data has been backed up before proceeding.

Important

The partition sizes shown throughout this guide reflect the hardware used in our deployment. Adjust all sizes to match your own storage capacity and retention requirements.

Symptoms

You may observe one or more of the following:

Internal Server Error within the web interface
/var repeatedly reaches 100% utilization
Metadata storage grows unexpectedly
Packet capture continues while the GUI becomes unstable
System health degrades during traffic spikes

Step 1 – Verify the Existing Storage Layout

Begin by reviewing the current storage configuration.

lsblk

A typical appliance should resemble the following layout.

sda
├── /boot
├── /var
├── /
├── /ds
├── /gui
└── /home

sdb
└── indexVG/indexLV
    └── /var/lib/solera/meta

sdc
└── captureVG/captureLV
    └── /pfs

Your hardware may differ, but separate logical volumes for metadata and packet storage should exist.

In our environment, only captureVG existed. The metadata volume (indexVG) had never been created.

Step 2 – Remove the Existing Storage Configuration

Delete the existing partitions from the storage device.

Example:

fdisk /dev/sdb

Delete each partition before writing the changes.

d
w

Remove any existing volume groups.

vgdisplay
vgremove indexVG
vgremove captureVG

Step 3 – Initialize the Disk

Large storage devices may require GPT before partitions larger than 2 TB can be created.

parted /dev/sdb

Example:

mklabel gpt
yes
unit TB
mkpart primary 0 0
print
quit

Step 4 – Create the Partitions

Recreate the storage layout.

fdisk /dev/sdb

Create two Linux LVM partitions.

Example sizes used in our deployment:

Partition	Purpose	Size
sdb1	Metadata	4 TB
sdb2	Packet Capture	36 TB

Adjust these values to match your environment.

Step 5 – Create Volume Groups

Create separate volume groups.

vgcreate indexVG /dev/sdb1
vgcreate captureVG /dev/sdb2

Verify the volume groups.

vgdisplay

Step 6 – Create Logical Volumes

Create the logical volumes.

lvcreate -L 3.9T -n indexLV indexVG
lvcreate -L 36T -n captureLV captureVG

Again, adjust the sizes as appropriate for your hardware.

Step 7 – Format and Mount the Volumes

Format each logical volume.

mkfs.xfs /dev/mapper/indexVG-indexLV
mkfs.xfs /dev/mapper/captureVG-captureLV

Mount the filesystems.

mount /dev/mapper/indexVG-indexLV /var/lib/solera/meta
mount /dev/mapper/captureVG-captureLV /pfs

Step 8 – Update `/etc/fstab`

Verify the required mount entries exist.

At minimum, ensure entries similar to the following are present.

/dev/captureVG/captureLV    /pfs                  xfs   defaults,noatime,nodiratime,nobarrier,noauto,nosuid,nodev,allocsize=64m,nofail
/dev/indexVG/indexLV        /var/lib/solera/meta xfs   defaults,noatime,nodiratime,nobarrier,noauto,nosuid,nodev,allocsize=8m,nofail

If these entries are missing, add them before rebooting.

Step 9 – Mount and Reboot

Mount all configured filesystems.

mount

Reboot the appliance.

reboot

Verification

After the appliance starts:

Verify the storage layout.

lsblk

Verify the volume groups.

vgdisplay

In our environment, the operating system reassigned disk names after the reboot.

For example:

Original sdb became sda
Original sda became sdb

This is expected on some hardware platforms.

Confirm:

/pfs is mounted.
/var/lib/solera/meta is mounted.
Packet capture resumes normally.
Metadata generation resumes normally.
/var no longer grows uncontrollably during periods of heavy traffic.