1TB software RAID 5 with one hot spare.

Impotence · 23-07-2008, 03:49 AM

I have been putting this off for a month, due to worrying too much about doing it wrong... i would appreciate it if someone would take me throgh this step by step.

Distribution

Kubuntu 8.04 KDE4 remix (x86)

Hardware

3 x 500GB SATA drives (sdc,sdd and sde) [unformatted, unmounted] to be RAID 5'd.
1 x 500GB SATA drive (sdb) [ext3, /home, 99% full] for a hot spare.

The hot spare drive cannot be added until the array has been built as i do not have anywhere to store it (and tested to death, I'm not wanting to loose 430GB of data!)

What i have done so far

Tied up SATA data & power cables to stop them from moving
Used hot glue to prevent sata data & power connectors from coming lose from HDD / Motherboard
Installed mdadm

What i need to do (help!)

Create the RAID
Setup RAID monitoring [audible alarm if a disk dies]
create a lvm on it
format the lvm (Suggestions? something journaled?)
Create test data on the RAID (a copy of /home)
Selectively drop individual disks out of the array (can't unplug them, dban them?) and learn to rebuild it.
when i am confident that it is 'stable' mount it as /home
Add the old /home disk (sdb) to the array as a hot spare
Possibly use sdb to make a 1.5TB raid once i have filled 1TB

Questions

If i change distribution / when i do a clean install, how do i mount the raid? what files (if any) do i have to have a copy of to do it (such as the mdadm config file?)

Impotence · 30-07-2008, 07:30 PM

I have managed to do it myself, my thanks to anyone who took the time to read my post but where unable to help.

I am currently writing up what i have learned in the form of a idiot proof distribution independent tutorial... I will post a link when it is complete.

I also plan to fill in https://wiki.ubuntu.com/LVMOnRaid when i have finished!

Anti-Trend · 30-07-2008, 11:43 PM

Sorry, didn't see this thread until just now. Glad you got it figured out though. On the bright side, I'm sure you learned more from researching it yourself anyway.

Impotence · 31-07-2008, 06:16 AM

yeah, just one thing that i want to double check though.

When creating the lvm, i used the formula below (and then rounded it up to 2,4,8,16,32 or 64) to calculate the physical extent size, for a 2TB raid i worked out that i should set it to 32MB (rounded up from 30.048)... is this right?

Code:

RAID-SIZE*1000000000/1024/1024/65000

Anti-Trend · 31-07-2008, 06:08 PM

Looks right to me. For simplicity's sake, this is more or less the process I used for my own software RAIDs in Linux: http://linuxhelp.blogspot.com/2005/0...-in-linux.html

One thing to note however is that there are advantages and disadvantages of running an LVM on top of a RAID rather than simply formatting the RAID directly. If you run an LVM on top of the RAID, you will be unable to use the journalling capabilities of your filesystem. In other words, I hope you have a UPS.

That said, the LVM makes it quite a simple matter to grow or shrink your filesystem should you want to change the amount of devices allocated for this purpose. So the bottom line is this: If you want to preserve the robustness of your journalled filesystem, don't do LVM at all, just format the RAID and mount it directly. If scalability is more important, go with LVM.

Impotence · 31-07-2008, 10:55 PM

I spent nearly an hour in #LVM on freenode today trying to get this figured out...

Turns out the PE limit was 65,535 (not 65,000) and that lvm2 has no limit on the number of PE's! So now I'm left trying to decide on a good size without a lot to base it on.

Could you explain why LVM renders journals useless? I'm having a hard time trying to work out why... and unfortunately nobody i have asked knows enough about it to have heard about that problem :/

saying that, i have been thinking about a UPS for awhile i just can't afford the dam things, just finished paying off a £500/$1000 overdraft that i blew on beer and pizza... perhaps i could spend it on something worthwhile next time the cash goes to my head!

Anti-Trend · 31-07-2008, 11:36 PM

Quote:

Originally Posted by Impotence

Could you explain why LVM renders journals useless? I'm having a hard time trying to work out why... and unfortunately nobody i have asked knows enough about it to have heard about that problem :/

On closer look, this was a problem specific to EXT3 and it seems to have been fixed in 2.6.18 kernels and later, and possibly even backported into earlier kernels. I ran into that problem on RHEL 4, as it used a 2.6.9 kernel. XFS seems to uneffected by this issue in either case, though it will automatically disable write barriers if it detects that its running on LVM. See here:
SGI - Developer Central Open Source | XFS

Impotence · 01-08-2008, 12:29 AM

ok, good to know

I formatted it as ext3, i also set the reserved blocks to 0% as its going to be /home (reserving blocks for the super user seemed a bit stupid, the default 5% would also have taken 45GB :o).

Once this is done and documented all i have left to do is copy everything across and mount it as /home... I'm not looking forward to cleaning up the formatting, spelling & grammar in the tutorial, but I'm glad its almost finished

Anti-Trend · 01-08-2008, 08:21 PM

One thing to keep in mind with Ext3 is that without reserved blocks, it will fragment pretty badly when you get close to full capacity. That's one reason why I use XFS for my own 1.5TB array: I get the full capacity, and have a live defragmentation utility which I can cron to run monthly or so.

Impotence · 02-08-2008, 12:02 AM

If i remember correctly, fragmentation starts to become a problem on a ext3 partition when it is over 90% full. But I'm not too worried about it yet, I plan to add another TB to my raid and hopefully by the time i have filled that ext4 will be stable (which includes a online fragmentation utility and is backwards compatible with ext3).

I will put the reserved blocks as 1% in the tutorial, which should(?) give Linux enough to play with when people run out of disk space without consuming too much disk space (percentages are evil on large disks / RAIDs)

23-07-2008, 03:49 AM	#1 (permalink) Top
Impotence May the source be with u! Join Date: Dec 2004 Age: 19 Male Posts: 1,877 Times Helpful: 126 My Mood: Stressed Status: Offline My Computer	1TB software RAID 5 with one hot spare. I have been putting this off for a month, due to worrying too much about doing it wrong... i would appreciate it if someone would take me throgh this step by step. Distribution Kubuntu 8.04 KDE4 remix (x86) Hardware 3 x 500GB SATA drives (sdc,sdd and sde) [unformatted, unmounted] to be RAID 5'd. 1 x 500GB SATA drive (sdb) [ext3, /home, 99% full] for a hot spare. The hot spare drive cannot be added until the array has been built as i do not have anywhere to store it (and tested to death, I'm not wanting to loose 430GB of data!) What i have done so far Tied up SATA data & power cables to stop them from moving Used hot glue to prevent sata data & power connectors from coming lose from HDD / Motherboard Installed mdadm What i need to do (help!) Create the RAID Setup RAID monitoring [audible alarm if a disk dies] create a lvm on it format the lvm (Suggestions? something journaled?) Create test data on the RAID (a copy of /home) Selectively drop individual disks out of the array (can't unplug them, dban them?) and learn to rebuild it. when i am confident that it is 'stable' mount it as /home Add the old /home disk (sdb) to the array as a hot spare Possibly use sdb to make a 1.5TB raid once i have filled 1TB Questions If i change distribution / when i do a clean install, how do i mount the raid? what files (if any) do i have to have a copy of to do it (such as the mdadm config file?) __________________ Dear intel,Already Made My Decision Never trust a computer you can't repair yourself. Windows Vista - Viruses Intruders Spyware Trojans Adware Why Linux is Better!\|Pastafairianism\|Say no to National ID\|Firefox Extensions Guide scientology has been getting alot of attention recently When the rich wage war, it is the poor who die. Where are the WMD's that our governments used to justify the war? Last edited by Impotence; 23-07-2008 at 04:03 AM.

30-07-2008, 07:30 PM	#2 (permalink) Top
Impotence May the source be with u! Join Date: Dec 2004 Age: 19 Male Posts: 1,877 Times Helpful: 126 My Mood: Stressed Status: Offline My Computer	I have managed to do it myself, my thanks to anyone who took the time to read my post but where unable to help. I am currently writing up what i have learned in the form of a idiot proof distribution independent tutorial... I will post a link when it is complete. I also plan to fill in https://wiki.ubuntu.com/LVMOnRaid when i have finished! __________________ Dear intel,Already Made My Decision Never trust a computer you can't repair yourself. Windows Vista - Viruses Intruders Spyware Trojans Adware Why Linux is Better!\|Pastafairianism\|Say no to National ID\|Firefox Extensions Guide scientology has been getting alot of attention recently When the rich wage war, it is the poor who die. Where are the WMD's that our governments used to justify the war?

30-07-2008, 11:43 PM	#3 (permalink) Top
Anti-Trend Nonconformist Geek Join Date: Oct 2003 Age: 27 Male Posts: 4,813 Times Helpful: 524 Status: Online My Computer	Sorry, didn't see this thread until just now. Glad you got it figured out though. On the bright side, I'm sure you learned more from researching it yourself anyway. __________________ See My Desktop \| Try Linux \| Install Linux \| Visit My Server \| Support Hardware Forums \| Easily rip DVDs in Linux

31-07-2008, 06:16 AM	#4 (permalink) Top
Impotence May the source be with u! Join Date: Dec 2004 Age: 19 Male Posts: 1,877 Times Helpful: 126 My Mood: Stressed Status: Offline My Computer	yeah, just one thing that i want to double check though. When creating the lvm, i used the formula below (and then rounded it up to 2,4,8,16,32 or 64) to calculate the physical extent size, for a 2TB raid i worked out that i should set it to 32MB (rounded up from 30.048)... is this right? Code: RAID-SIZE1000000000/1024/1024/65000 __________________ Dear intel,Already Made My Decision Never trust a computer you can't repair yourself.* Windows Vista - Viruses Intruders Spyware Trojans Adware Why Linux is Better!\|Pastafairianism\|Say no to National ID\|Firefox Extensions Guide scientology has been getting alot of attention recently When the rich wage war, it is the poor who die. Where are the WMD's that our governments used to justify the war?

31-07-2008, 06:08 PM	#5 (permalink) Top
Anti-Trend Nonconformist Geek Join Date: Oct 2003 Age: 27 Male Posts: 4,813 Times Helpful: 524 Status: Online My Computer	Looks right to me. For simplicity's sake, this is more or less the process I used for my own software RAIDs in Linux: http://linuxhelp.blogspot.com/2005/0...-in-linux.html One thing to note however is that there are advantages and disadvantages of running an LVM on top of a RAID rather than simply formatting the RAID directly. If you run an LVM on top of the RAID, you will be unable to use the journalling capabilities of your filesystem. In other words, I hope you have a UPS. That said, the LVM makes it quite a simple matter to grow or shrink your filesystem should you want to change the amount of devices allocated for this purpose. So the bottom line is this: If you want to preserve the robustness of your journalled filesystem, don't do LVM at all, just format the RAID and mount it directly. If scalability is more important, go with LVM. __________________ See My Desktop \| Try Linux \| Install Linux \| Visit My Server \| Support Hardware Forums \| Easily rip DVDs in Linux

31-07-2008, 10:55 PM	#6 (permalink) Top
Impotence May the source be with u! Join Date: Dec 2004 Age: 19 Male Posts: 1,877 Times Helpful: 126 My Mood: Stressed Status: Offline My Computer	I spent nearly an hour in #LVM on freenode today trying to get this figured out... Turns out the PE limit was 65,535 (not 65,000) and that lvm2 has no limit on the number of PE's! So now I'm left trying to decide on a good size without a lot to base it on. Could you explain why LVM renders journals useless? I'm having a hard time trying to work out why... and unfortunately nobody i have asked knows enough about it to have heard about that problem :/ saying that, i have been thinking about a UPS for awhile i just can't afford the dam things, just finished paying off a £500/$1000 overdraft that i blew on beer and pizza... perhaps i could spend it on something worthwhile next time the cash goes to my head! __________________ Dear intel,Already Made My Decision Never trust a computer you can't repair yourself. Windows Vista - Viruses Intruders Spyware Trojans Adware Why Linux is Better!\|Pastafairianism\|Say no to National ID\|Firefox Extensions Guide scientology has been getting alot of attention recently When the rich wage war, it is the poor who die. Where are the WMD's that our governments used to justify the war?

01-08-2008, 12:29 AM	#8 (permalink) Top
Impotence May the source be with u! Join Date: Dec 2004 Age: 19 Male Posts: 1,877 Times Helpful: 126 My Mood: Stressed Status: Offline My Computer	ok, good to know I formatted it as ext3, i also set the reserved blocks to 0% as its going to be /home (reserving blocks for the super user seemed a bit stupid, the default 5% would also have taken 45GB :o). Once this is done and documented all i have left to do is copy everything across and mount it as /home... I'm not looking forward to cleaning up the formatting, spelling & grammar in the tutorial, but I'm glad its almost finished __________________ Dear intel,Already Made My Decision Never trust a computer you can't repair yourself. Windows Vista - Viruses Intruders Spyware Trojans Adware Why Linux is Better!\|Pastafairianism\|Say no to National ID\|Firefox Extensions Guide scientology has been getting alot of attention recently When the rich wage war, it is the poor who die. Where are the WMD's that our governments used to justify the war?

01-08-2008, 08:21 PM	#9 (permalink) Top
Anti-Trend Nonconformist Geek Join Date: Oct 2003 Age: 27 Male Posts: 4,813 Times Helpful: 524 Status: Online My Computer	One thing to keep in mind with Ext3 is that without reserved blocks, it will fragment pretty badly when you get close to full capacity. That's one reason why I use XFS for my own 1.5TB array: I get the full capacity, and have a live defragmentation utility which I can cron to run monthly or so. __________________ See My Desktop \| Try Linux \| Install Linux \| Visit My Server \| Support Hardware Forums \| Easily rip DVDs in Linux

02-08-2008, 12:02 AM	#10 (permalink) Top
Impotence May the source be with u! Join Date: Dec 2004 Age: 19 Male Posts: 1,877 Times Helpful: 126 My Mood: Stressed Status: Offline My Computer	If i remember correctly, fragmentation starts to become a problem on a ext3 partition when it is over 90% full. But I'm not too worried about it yet, I plan to add another TB to my raid and hopefully by the time i have filled that ext4 will be stable (which includes a online fragmentation utility and is backwards compatible with ext3). I will put the reserved blocks as 1% in the tutorial, which should(?) give Linux enough to play with when people run out of disk space without consuming too much disk space (percentages are evil on large disks / RAIDs) __________________ Dear intel,Already Made My Decision Never trust a computer you can't repair yourself. Windows Vista - Viruses Intruders Spyware Trojans Adware Why Linux is Better!\|Pastafairianism\|Say no to National ID\|Firefox Extensions Guide scientology has been getting alot of attention recently When the rich wage war, it is the poor who die. Where are the WMD's that our governments used to justify the war? Last edited by Impotence; 02-08-2008 at 12:13 AM.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
What to do with spare PC?	RockDiesel	General Hardware	7	13-07-2008 12:51 AM
Nvidia MediaSheild RAID and Intel Storage Matrix RAID compatibility	aalaap	CPU, Motherboards and Memory	0	24-06-2008 11:25 AM
Spare SLI Bridges	RangerMoose	CPU, Motherboards and Memory	5	05-01-2007 03:09 AM
If you have got some old spare parts	Crusha19	Buy / Sell / Trade	3	09-11-2006 06:47 PM
software RAID5 on a PCI card supporting RAID 0+1 ?	markey164	Storage Devices	3	22-08-2006 03:49 PM