Linux IDE-RAID Notes - IDE-3WRAID
# df -H /mnt/tmp/md*
Filesystem
Size Used Avail Use% Mounted on
/dev/md0
530G 0.0G 530G 0% /mnt/tmp/md0
/dev/md1
519G 0.0G 519G 0% /mnt/tmp/md1
# df -h /mnt/tmp/md*
Filesystem
Size Used Avail Use% Mounted on
/dev/md0
493G 0.0G 493G 0% /mnt/tmp/md0
/dev/md1
483G 0.0G 483G 0% /mnt/tmp/md1
Suggested production configurations included:
| RAID configuration | Filesystem | Total Capacity | Comment |
| 2 software-RAID5 arrays of 8-drives |
|
|
less storage, more redundancy & throughput |
| 1 software-RAID5 array of 16-drives |
|
|
more storage, less redundancy & throughput |
There is a limit of 1TB for ext2, so reiserfs must be used for filesystems larger than 1TB. The performance measurements below show that reiserfs performance with this configuration drops unacceptably with very large arrays. Unfortunately, I had to put this system into service before I could look find a solution to this problem, perhaps the journal could be written to a separate spindle? So the production configuration is 2 half-TB ext2 arrays. The sizes of the file systems are slightly different because the IBM 75GXP drives seem to come in two sizes, 76.8GB and 75.3GB.
Important - Make sure to do a basic sanity check each drive using "hdparm -Tt" and check for expected single drive performance. Repair as neccessary by reseating cables, swapping cables, drives and controllers. I have seen some drives with significantly slower performance! This could save you a lot of time, as poorly performing hardware invalidates the whole test suite. It's a good idea to add "hdparm -Tt" to your test scripts as a sanity check on each drive before JBOD or RAID tests.
Related note - Bart Locanthi also reported slow motherboard controller configuration (UDMA/33 versus UDMA/100) due to BIOS settings, so check to make sure that both your primary/secondary motherboard controllers and related master/slave channels are enabled in your BIOS, e.g., with the setting "auto", not "none".
I much prefer the Antec 18-bay case, but it is discontinued. Any server case with a drive compartment on the back side of the motherboard will probably have serious cabling problems. Minimizing drive-cable lengths while maintaining adequate access to expansion cards and drive bays may be a challenge, but I'm still hoping to find a satisfactory case that is in production -- help! The makeshift CDROM-case bracket may help to reduce drive bay requirement. I'm somewhat astonished at the lack of attention to high-density low-cost (IDE-drive) storage solutions. One new product is the Amax Terabyte RAID server solution.
The Seventeam power supply has only 3 power cables. Instead, I recommend the Enlight EN-8407362 ATX 400W single power supply for server cases which has 6 power cables.
Also, please make sure that there is no small BIOS limit to the number of controllers in the system, e.g., not the limite of 3 Promise controllers per system. Where inexpensive high-density storage is the goal, we can picture a system with all slots filled to capacity, mostly with 3ware controllers! With the above layout fix for cabling, this would enable something like the Amax Terabyte RAID server with 32 drives but with 4 controllers on one motherboard. Even more could be fit onto a server motherboard. If only ... connector fix ... good server case ... power supply ... motherboard, e.g., Asus CUR-DLS with built-in video and ethernet plus 7 PCI slots $599 (ouch) ... yeah, dream on ...
# ed /etc/sysconfig/harddisks
USE_DMA=1
MULTIPLE_IO=16
EIDE_32BIT=1
LOOKAHEAD=1
EXTRA_PARAMS=
# ed /etc/rc.d/rc.sysinit
disk[0]=s; disk[1]=hda; disk[2]=hdb; disk[3]=hdc; disk[4]=hdd;
disk[5]=hde; disk[6]=hdf; disk[7]=hdg; disk[8]=hdh;
disk[9]=hdi; disk[10]=hdj; disk[11]=hdk; disk[12]=hdl;
disk[13]=hdm; disk[14]=hdn; disk[15]=hdo; disk[16]=hdp;
disk[17]=hdq; disk[18]=hdr; disk[19]=hds; disk[20]=hdt;
if [ -x /sbin/hdparm ]; then
for device in 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
17 18 19 20; do
Reiserfs tools - update required -
http://www.namesys.com/
# tar xf reiserfsprogs-3.x.0f.tar
# cd reiserfsprogs-3.x.0f
# ./configure; make all install
# VER=2.4.2
# umask 002
# mkdir /usr/src/linux-$VER; cd /usr/src/linux-$VER; tar xf linux-$VER.tar;
mv linux/* .; rmdir linux
# cd ..; rm /usr/src/linux; ln -s /usr/src/linux-$VER /usr/src/linux
# cd linux-$VER
# make mrproper
# make xconfig # (remember to enable any other drivers
for SCSI support, Network device support, Sound, etc)
Code maturity level options
y Prompt for development and/or
incomplete code/drivers
Multi-device support (RAID and LVM)
y Multiple devices driver support
(RAID and LVM)
y RAID
support
y
Linear (append) mode
y
RAID-0 (striping) mode
y
RAID-1 (mirroring) mode
y
RAID-4/RAID-5 mode
ATA/IDE/MFM/RLR support
IDE, ATA and ATAPI Block
devices
y Generic PCI bus-master DMA support
y Use PCI DMA by default when
available
y Intel
PIIXn chipsets support
y
PIIXn Tuning support
y PROMISE
PCD PDC20246/PDC20262 support
y
Special UDMA Feature
SCSI support
SCSI low-level drivers
y 3ware Hardware ATA-RAID support
Network Device Support
Ethernet (10 or 100Mbit)
(don't forget the network card)
File Systems
y Reiserfs support
Network File Systems
y NFS file system support
y Provide
NFSv3 client support
y NFS server support
y Provide
NFSv3 server support
y SMB file system support (to
mount Windows shares etc.)
# make dep clean bzImage modules modules_install
# sh scripts/MAKEDEV.ide
# cp arch/i386/boot/bzImage /boot/vmlinuz-$VER
# cp System.map /boot/System.map-$VER
# ed /etc/lilo.conf
image=/boot/vmlinuz-2.4.2
label=linux
read-only
root=/dev/hda5
# lilo # LILO
mini-HOWTO, BootPrompt-HowTo
# reboot
For bonnie++ text results, see ide-3wraid.txt.
| Configuration | total
drives |
capacity | max-GB/
8-drives |
comment |
| JBOD |
|
|
600
|
test 1 to 8 drives |
| RAID0 (stripe) |
|
|
600
|
test 2- to 8-drive arrays |
| RAID1 (mirror) |
|
|
300
|
test 1 to 4 * 2-drive arrays |
| RAID10 (stripe of mirror) |
|
|
300
|
test 4- or 8-drive array |
| RAID5 |
|
|
525
|
test 3- to 8-drive arrays |
Performance may be affected by PCI slots and interrupts. Tests with different PCI slots showed similar performance, at least with this motherboard. The different configurations also had diverse interrupt assignments, so interrupts can probably be overlooked for now. So PCI slots 1 and 3 were chosen for cabling reasons. This also leaves PCI slot 5 available for another controller if I can figure out how to mount 8 more drives in the cabinet.
A chunk size of 64 is used uniformly for these tests.
Clarification:
cyan is 3w ext2 RAID10 (circa 1.3MBps)
yellow is 3w ext2 RAID5
Comments:
JBOD1 - single controller, looks OK
JBOD2 - two controllers, additional throughput with
second controller, looks OK
3w ext2 RAID0 - 60MBps read speed is lower than
desired with 64KB stripe (consider stripe up to 1MB)
3w ext2 RAID1 - suggests single disk reads
3w ext2 RAID10 - 1.3MBps is hobbling (stripe of
1MB had no effect) - something is broken
3w ext2 RAID5 - like RAID0
3w reiserfs RAID5 - looks OK
2.4.2 ext2 RAID0 - looks reasonable
2.4.2 reiserfs RAID0 - works for 16 drives (> 1TB),
but performance drops significantly
2.4.2 ext2 RAID5 - looks reasonable, some slowdown
with 13-15 drives
2.4.2 reiserfs RAID5 - works for 16 drives (> 1TB),
but performance drops unacceptably for 14-16 drives
Something is clearly wrong with 3w RAID10. One additional note is that 3w RAID1 did drive calibration, but 3w RAID10 did not.
| key & configuration | Bonnie read
MB/sec |
Bonnie write
MB/sec |
Comment |
| PIO ex. I34GXP | 4.1 | 4.3 | Promise Ultra66 |
| I75GXP I66 | 36.5 | 26.9 | Intel CC820 ICH |
| I75GXP P100 | 36.5 | 29.4 | Promise Ultra100 |
| I75GXP P100 ReiserFS | 35.8 | 35.4 | Promise Ultra100 |
| I75GXP 3W |
36.6
|
36.4
|
3ware Escalade 6800 |
| I34RAID | 66.8 | 35.6 | Promise Ultra66 |
| M40RAID | 46.6 | 35.5 | mixed controllers |
| S18RAID | 39.5 | 36.7 | 2940U2W W/LW mix |
| 3WRAID | 62.5 | 30.4 | 3ware Escalade 6800 JBOD (SW RAID5) |
Explanation for the above, in order of test:
PIO ex. I34GXP - PIO reference
I75GXP I66 - Intel PIIX4 reference
I75GXP P100 - Promise Ultra100 reference
I34RAID, M40RAID, S18RAID - reference
| Sequential Output | Sequential Input | Random | |||||||||||
| Per Char | Block | Rewrite | Per Char | Block | |||||||||
| Machine | MB | K/sec | %CPU | K/sec | %CPU | K/sec | %CPU | K/sec | %CPU | K/sec | %CPU | /sec | %CPU |
| PIO ex. I34GXP | 500 | 2845 | 64.1 | 4315 | 50.5 | 2053 | 10.5 | 2743 | 32.0 | 4114 | 5.4 | 86.9 | 2.1 |
| I75GXP I66 | 500 | 9602 | 97.2 | 26937 | 17.7 | 14687 | 19.8 | 9727 | 93.8 | 36482 | 21.2 | 155.8 | 1.7 |
| I75GXP P100 | 500 | 9629 | 97.5 | 29428 | 20.5 | 15312 | 19.5 | 9798 | 94.9 | 36462 | 21.7 | 158.4 | 1.7 |
| I75GXP P100 ReiserFS | 500 | 7972 | 98.2 | 35377 | 65.1 | 15243 | 24.9 | 8509 | 93.0 | 35762 | 27.2 | 148.8 | 2.5 |
| I34RAID | 500 | 7251 | 91.9 | 35571 | 30.2 | 18232 | 35.0 | 8134 | 95.9 | 66774 | 46.8 | 207.6 | 3.0 |
| M40RAID | 500 | 7443 | 91.3 | 35546 | 29.5 | 17707 | 34.0 | 8251 | 95.4 | 46554 | 32.6 | 322.3 | 4.4 |
| S18RAID | 500 | 4857 | 98.3 | 39451 | 78.8 | 16078 | 55.2 | 6533 | 95.0 | 36652 | 35.6 | 495.8 | 11.8 |
| 3WRAID | 4000 | 11770 | 85 | 30398 | 13 | 21990 | 20 | 11050 | 82 | 62470 | 49 | 245.1 | 1 |