Linux IDE-RAID Notes

How to build an inexpensive large (terabyte) RAID

Terabyte RAID testing is in progress, as performance is questionable. 

# df /mnt/tmp
Filesystem           1k-blocks      Used   Available Use% Mounted on
/dev/md0             1050858548     32840 1050825708   1% /mnt/tmp
Name Size Date Hard Drives Controllers Details
I75RAID 1.1 TB 2001-??? 16 IBM 75GXP 3 Promise Ultra100 + Intel 820 ide-i75raid
3WRAID 530+519 GB 2001-Mar 16 IBM 75GXP 2 3ware Escalade 6800 ide-3wraid
M40RAID 282 GB 2000-Jul 8 Maxtor 40 3 Promise Ultra66 + Intel BX440-2 ide-m40raid
I34RAID 236 GB 2000-Feb 8 IBM 34GXP 2 Promise Ultra66 ide-i34raid
S18RAID 167 GB 2000-Aug 11 Seagate 18 Adaptec 2940U2W scsi-s18raid

Click on the "details" links above for hardware and software (kernel and patch) configuration, plus other notes.

Why IDE-RAID(5)?

IDE drives are 2 to 4 times less expensive than equivalent capacity SCSI drives, and with Ultra ATA/66 and Ultra ATA/100, the performance has improved.  The Promise Technology Ultra100 and Ultra66 controller cards are inexpensive.  Linux has software RAID.  So can we build an affordable large RAID to address our growing demands for streaming music and video?

Yes, it is possible, but only with the right combination of pieces.  After a lot of surfing and testing of various kernels and patches, here's the scoop.  Current IDE-RAID hardware is limited to RAID 0/1/10 and four (ex., Promise FastTrack) or eight (ex., 3ware Escalade) drives, so Linux software RAID is the way to go for more drives.  The recommended new software RAID has been included since RedHat 6 and is in recent kernels.  Support for the Promise controller cards (including Ultra100) is available as a patch for 2.2 and is already included in 2.4.

In order of importance, starting with the most important, the goals are:

  1. minimum cost (ex., IDE instead of SCSI)
  2. maximum capacity (in a single large volume)
  3. reliability
  4. performance
So RAID5 is the obvious choice and focus.

A major issue to be addressed is the file-system type.  The overhead for ext2 grows with increasing file-system size, doing a fsck on a large file system takes a long time.  I'm testing reiserfs with 2.4 (journaling FS on RAID5 is not safe in 2.2), and plan to look into xfs when it is more stable.  At present, Linux 2.4.0 still has performance problems related to IDE and RAID5.  See the ide-i75raid page for the details.

This document is also a history of large RAID systems that I have constructed, tested, and put into service.

IDE Hard Drives

Click heading-link above to view a table of current IDE hard drives of greater than 40 GB capacity.

Recommended IDE Controllers

Promise Technology
    Ultra100 - PCI Ultra ATA/100 Controller Card $35
    Ultra66 - PCI Ultra ATA/66 Controller Card $21
    connector away from face plate is channel 0, PCI slot toward AGP is controller 0

Selected Disk Specifications

key vendor name model GB RPM Seek ms interface IF/peak MB/sec Media/sus. MB/sec
I34GXP IBM Deskstar34GXP DPTA-373420 34.2 7400 9.0 IDE Ultra ATA/66 66 22.9
I36XP IBM Ultrastar36XP DRHS36V 36.4 7200 7.5 SCSI Ultra2(Wide) LVD 80 19.7
I36ZX IBM Ultrastar36ZX DMVS36V 36.7 10000 4.9 SCSI Ultra2(Wide) LVD 80 29.5
I75GXP IBM Deskstar75GXP DTLA-307075 75 7200 8.5 IDE Ultra ATA/100 100 37.0
MDM40  Maxtor DiamondMax40 94098U8 41.0 5400 9.0 IDE Ultra ATA/66 66 36.9
MDMP40 Maxtor DiamondMaxPlus40 54098U8 41.0 7200 9.0 IDE Ultra ATA/66 66 43.2
QA318S Quantum Atlas III 18SN QM318000TD 18.2 7200 7.8 SCSI Ultra-SE(Narrow) 20 12.0
SB18W Seagate Barracuda 18W ST118273W 18.2 7200 7.9 SCSI Ultra Wide 40 16.0
SB18WL Seagate Barracuda 18LW ST118273LW 18.2 7200 7.9 SCSI Ultra2(Wide) LVD 80 16.0
WD102 Western Digital Caviar 5400 WD102AA 10.2 5400 9.5 IDE Ultra ATA/66 66 29.5

Performance Summary

key & configuration Bonnie read
MB/sec		 Bonnie write
MB/sec		 Comment
PIO ex. I34GXP 4.1 4.3 Promise Ultra66
I34GXP P66 22.3 15.0 Promise Ultra66
I36XP SCSI U2W 17.6 6.0 Adaptec 2940U2W
I36ZX SCSI U2W 24.3 8.2 Adaptec 2940U2W
I75GXP I66 36.5 26.9 Intel 820 ICH
I75GXP P100 36.5 29.4 Promise Ultra100
I75GXP 3W 36.6 36.4 3ware Escalade 6800
MDMP40 I33 22.4 28.6 Intel 440BX-2 PIIX4E
MDMP40 I66 25.9 31.5 Intel CC820
MDMP40 P66 28.6 31.9 Promise Ultra66
QA318S SCSI SN 11.7 13.0 Adaptec 2940U2W
SB18W SCSI W 16.9 14.8 Adaptec 2940U2W
SB18LW SCSI LW 17.1 14.8 Adaptec 2940U2W
WD102 I66 15.7 24.3 Intel CC820 ICH
WD102 P66 11.8 24.7 Promise Ultra66
I34RAID  66.8  35.6 Promise Ultra66 
M40RAID 46.6 35.5 Promise Ultra66 + Intel 440BX-2
S18RAID 39.5 36.7 2940U2W W/LW mix
3WRAID 62.5 30.4 3ware 6800 JBOD (SW RAID5)
I75RAID 45.2 5.4 Promise Ultra100 + Intel 820

hdparm

    http://www.oreillynet.com/pub/a/linux/2000/06/29/hdparm.html
    # top
    # hdparm -Tt /dev/hde # measure device reads, run at least 3x

Bonnie

    http://www.textuality.com/bonnie/ -file-system performance benchmarks
    # Bonnie -s 500 -html -m ...

bonnie++

    http://sourceforge.net/projects/bonnie/ - new location, plus new Experimental version
    http://www.coker.com.au/bonnie++/ - old location
    # bonnie -p 2 -u root
    # for i in 1 2; do bonnie -y -u root -s 500 -m ... & done
 
Sequential Output Sequential Input Random
Per Char Block Rewrite Per Char Block
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
PIO ex. I34GXP 500 2845 64.1 4315 50.5 2053 10.5 2743 32.0 4114 5.4 86.9 2.1
I34GXP P66 500 7789 98.8 14983 12.6 7739 13.5 7909 92.6 22316 12.9 132.8 1.7
I36XP 500 3668 46.0 5979 6.5 3902 8.8 7072 83.2 17582 13.8 98.1 1.7
I36ZX 500 6408 81.1 8249 6.4 5281 8.8 8322 97.2 24290 15.2 154.9 2.3
I75GXP I66 500 9602 97.2 26937 17.7 14687 19.8 9727 93.8 36482 21.2 155.8 1.7
I75GXP P100 500 9629 97.5 29428 20.5 15312 19.5 9798 94.9 36462 21.7 158.4 1.7
MDMP40 I33 500 8050 99.1 22443 18.8 11489 19.9 8234 94.5 28648 17.0 136.4 1.4
MDMP40 I66 500 9588 97.8 31489 21.3 12580 22.7 9386 90.6 25934 14.0 132.7 1.8
MDMP40 P66 500 9457 97.6 31920 21.8 12668 26.2 9776 95.3 28582 19.2 137.6 2.1
QA318S 500 6922 90.2 12990 15.4 4020 9.1 6912 82.5 11659 9.4 68.9 1.2
SB18W 500 6367 97.7 16935 18.5 6785 16.1 6347 91.0 14805 12.5 204.9 3.9
SB18LW 500 6486 99.6 17141 19.0 6766 16.1 6383 91.3 14800 12.5 190.8 3.0
WD102 I66 500 9650 98.3 24314 15.1 8287 19.0 8980 86.7 15687 11.4 109.7 1.3
WD102 P66 500 9643 98.7 24724 15.7 8523 17.5 9455 91.3 11793 6.4 115.2 1.3
I34RAID 500 7251 91.9 35571 30.2 18232 35.0 8134 95.9 66774 46.8 207.6 3.0
M40RAID 500 7443 91.3 35546 29.5 17707 34.0 8251 95.4 46554 32.6 322.3 4.4
S18RAID 500 4857 98.3 39451 78.8 16078 55.2 6533 95.0 36652 35.6 495.8 11.8
3WRAID 4000 11770 85 30398 13 21990 20 11050 82 62470 49 245.1 1
I75RAID 500 5594 75 5434 16 4499 11 6816 83 45164 79 18.6 0

Recommended Miscellaneous Hardware

Antec KS011BX - ATX 18-bay black tower server case w/o power supply $287 (discontinued)
Antec 761345-77055-2 - 3 pin ball bearing fan - 92mm $9
Enlight EN-8407362 - ATX 400W single power supply for server cases $97
Enlight EN-89500SX4 - 9-bay server case, 400W power supply, ATX $226
CTG (Cables To Go) 18729 - 24in int Ultra DMA/ATA ribbon 3 connector IDE 33/66 UDMA $11

Other Interesting Hardware

Promise SuperTrak100 - 6-channel / 6-drive Ultra ATA/100 RAID 3&5 Card, $406
3ware Escalade 6800 - 8-channel / 8-drive Ultra ATA/66 RAID 0/1/10/5 controller, $357
Arco DupliDisk - IDE RAID1 transparent mirroring hardware
Adaptec AAA-UDMA RAID - 4-channel / 8-drive Ultra ATA/66 RAID 0/1/5, $375 (no linux.adaptec.com support, yet)

BIOS limitations for drives greater than 40GB

The BIOS may need to be updated for older motherboards and IDE controllers in order to support drives greater than 32GB in size.

Maxtor Information Bulletin - 32 GB BIOS Limitations
http://www.uwsg.indiana.edu/hypermail/linux/kernel/0003.3/0129.html

Disk Reliability

ftp://ftp.bitmover.com/pub/lmdd.tar.gz
LMDD(8) - LMBENCH man page
message on reliability test usage
example usage:
    write pattern:
        # /home/linux/opt/lmdd/lmdd of=XXX bs=10m move=1g opat=1
    read/check pattern:
        # /home/linux/opt/lmdd/lmdd if=XXX bs=10m move=1g ipat=1

References

http://www.fujitsu.co.jp/hypertext/ssworld/function/raid/raid_e.html - simple RAID intro with nice diagrams
http://www.linuxdoc.org/FAQ/Linux-RAID-FAQ/ - Linux-RAID Frequently Asked Questions
http://unthought.net/Software-RAID.HOWTO/ - The Software RAID How-to
http://people.redhat.com/mingo/raid-patches/ - software RAID patches
http://people.redhat.com/mingo/raidtools/ - RAID tools
http://www.kernel.org/pub/linux/kernel/people/hedrick/ - ide patches, including Promise
http://www.namesys.com/ - ReiserFS journaling FS

http://www.uwsg.iu.edu/hypermail/linux/kernel/ - Linux-Kernel mailing-list archive
http://marc.theaimsgroup.com/?l=linux-raid - Linux-RAID mailing-list archive
http://lists.omnipotent.net/reiserfs/  ReiserFS mailing-list archive
http://www.linuxdoc.org/HOWTO/Boot+Root+Raid+LILO.html - Boot+Root+RAID+LILO How-to
http://www.best.com/~cae/irqtune/ - IRQTUNE -- A Linux IRQ Priority Optimizer

http://linuxgazette.com/issue55/florido.html - "Journal File Systems" article from Linux Gazette issue 55
http://olstrans.sourceforge.net/release/OLS2000-ext3/OLS2000-ext3.html - ext3 (ext2 compatible) journaling FS
ftp://ftp.kernel.org/pub/linux/kernel/people/sct/ext3 - ext3 source
http://oss.sgi.com/projects/xfs/ - XFS journalling file system from SGI, beta
http://innominate.org/~phillips/ - tux2 journalling file system
mailto:majordomo@vger.kernel.org - subscribe linux-raid

http://linas.org/linux/raid.html - RAID solutions for Linux
http://www.linuxdoc.org/HOWTO/mini/Ultra-DMA-5.html - The Linux Ultra-DMA Mini-Howto: Off-board PCI UDMA interfaces
http://www.darmon.de/hardware/hardware.html - how to turn a Promise Ultra66/100 into a FastTrak66/100 (German)
http://www.anandtech.com/showdoc.html?i=1197 - Ultra ATA/66 Hard Drive Roundup
 

http://www.raidweb.com/ide.html - RAIDWeb By Electonix - IDE RAID Units
http://www.raidzone.com/ - RAIDZONE - High performance disk arrays using Ultra ATA disks

http://www.servercase.com/ - see advanced level server cases
http://www.jinco.com/ - cases, including file server cases
http://www.enhance-tech.com/ - see Products: Server Chassis
http://www.pricewatch.com/ - see CPU: Cases

NoBell Home - gjm - last update 2/20/2001