Networking

Unix and Linux network configuration. Multiple network interfaces. Bridged NICs. High-availability network configurations.

Applications

Reviews of latest Unix and Linux software. Helpful tips for application support admins. Automating application support.

Data

Disk partitioning, filesystems, directories, and files. Volume management, logical volumes, HA filesystems. Backups and disaster recovery.

Monitoring

Distributed server monitoring. Server performance and capacity planning. Monitoring applications, network status and user activity.

Commands & Shells

Cool Unix shell commands and options. Command-line tools and application. Things every Unix sysadmin needs to know.

Home » Disks and Volumes, Featured

Highly redundant array configuration

Submitted by on May 8, 2008 – 3:58 pm 2 Comments

The cost of storage arrays is falling along with the quality of their manufacture. To quote Lev Andropov from the “Armageddon”: “Components. American components, Russian Components, ALL MADE IN TAIWAN!” What in the nineties used to be quality storage hardware from Sun, these days is a bunch of consumer-quality hard drives thrown together into a poorly designed array enclosure.

For certain super-critical applications customers are expecting absolute data integrity and around-the-clock availability of their new pricey storage hardware. One way to solve the issue of low hardware quality and lousy vendor support is to throw money at the problem and to design a highly redundant storage system. Such a design would waste lots of space but it would ensure that data is accessible no matter what components fail.

RAID-5 is good but in the past I’ve lost three drives in a single RAID-5 set on a Sun 3510 in a matter of hours, leading to downtime and data loss. It turned out that there was a defective batch of drives. In a situation like this, when one drive goes, the RAID set starts rebuilding to a spare drive, generating lots of disk activity and hitting those bad blocks on the defective drive. It’s like a chain reaction. HA clusters – either Sun or VCS – are very impressive, but they are convoluted, expensive and, when they do fail, they are difficult to recover.

A Case Study

I had a pair of Sun 6140s and a pair of Sun v1280s – one production and one stand-by. I needed a design that ensured data and application availability regardless of what hardware component decided to fail: a disk, a bunch of disks, a NIC, both FC cards, a fiber cable, a tray, an entire array, or a server. Each array provided far more disk space than I needed, so I was in a position to waste disk space in exchange for redundancy. I never liked Sun HA Cluster or VCS – too complicated, too unreliable – failing over for a reason or without one – and requiring too much time to recover when the HA software screws up. Here’s a design I came up with:


To summarize: each v1280 has two dual-channel FCs, each 6140 has two quad-channel controllers. The servers are connected to both arrays in a way that would ensure an alternative communication path regardless of a failed device or component. This is how the physical connections are arranged on the outside. As for internal configuration:

Each array has 16 disks. These are arranged in 7 LUNs, 2 disks each with two disks left as spares for the hardware RAID-5. One LUN in each array is reserved as hot spare for the volume manager to use. Each logical volume is based on two mirrored (via the volume manager) LUNs – one from each array. Essentially, arrays are mirror copies of each other.

Additionally, each volume is replicated twice on each server and mounted under different paths. Rsync is used to synchronize the volumes on a regular basis. Not the most advanced solution, considering that Veritas VM is used as well as hardware RAID, but rsync mirroring is the last line of defense, when everything else fails.

So what you get in the end is 1/6 of usable disk space. Wasteful? Very. But when there is ample funding and extreme availability requirements, a configuration like this may be the ticket. I had this system implemented in a production environment. Since then we lost hard drives, FC cards (those go on a regular basis), and once an entire disk array went under, when the primary power supply failed and the backup power supply did not take over because of some firmware problem. Still, there was no downtime and no data loss.

Print Friendly, PDF & Email

2 Comments »

Leave a Reply

%d bloggers like this: