Back in early 2006 md/raid5 gained the ability to increase the number of devices in a RAID5, thus making more space available. As you can imagine, this is a slow process as every block of data (except possibly those in the first stripe) needs to be relocated. i.e they need to be read from one place and written to another. md/raid5 allows this reshaping to happen while the array is live. It temporarily blocks access to a few stripes at a time while those stripes a rearranged. So instead of the whole array being unavailable for several hours, little bits are unavailable for a fraction of a second each.
Then in early 2007 we gained the same functionality for RAID6. This was no more complex than RAID5, it just involved a little more code and testing.
Now, in mid 2009, we have most of the rest of the reshaping options that had been planned. These include changing the stripe size, changing the layout (i.e. where the parity blocks get stored) and reducing the number of devices.
Changing the layout provides valuable functionality as it is an important part of converting a RAID5 to a RAID6.
# How Level Changing Works
# Complexities of re-striping data
# Reducing the number of devices
# Currently supported reshapes
# Future Work
# Comments...
How Level Changing Works
If we think of "RAID5" as a little more generic than the standard definition, and allow it to be any layout which stripes data plus 1 parity block across a number of devices, then we can think of RAID4 as just a special case of RAID5. Then we can imagine a conversion from RAID0 to RAID5 as taking two steps. The first converts to RAID5 using the RAID4 layout with the parity disk as the last disk. This clearly doesn't require any data to be relocated so the change can be instant. It creates a degraded RAID5 in a RAID4 layout so it is not complete, but it is clearly a step in the right direction.I'm sure you can see what comes next. After converting the RAID0 to a degraded RAID5 with an unusual layout we would use the new change-the-layout functionality to convert to a real RAID5.
It is a very similar process that can now be used to convert a RAID5 to a RAID6. We first change the RAID5 to RAID6 with a non-standard layout that has the parity blocks distributed as normal, but the Q blocks all on the last device (a new device). So this is RAID6 using the RAID6 driver, but with a non-RAID6 layout. So we "simply" change the layout and the job is done.
A RAID6 can be converted to a RAID5 by the reverse process. First we change the layout to a layout that is almost RAID5 but with an extra Q disk. Then we convert to real RAID5 by forgetting about the Q disk.
Complexities of re-striping data
In all of this the messiest part is ensuring that the data survives a crash or other system shutdown. With the first reshape which just allowed increasing the number of devices, this was quite easy. For most of the time there is a gap in the devices between where data in the old layout in being read, and where data in the new layout is being written. This gap allows us to have two copies of that data. If we disable writes to a small section while it is being reshaped, then after a crash we know that the old layout still has good data, and simply re-layout the last few stripes from where-ever we recorded that we were up to.
This doesn't work for the first few stripes as they require writing the new layout over the old layout. So after a crash the old layout is probably corrupted and the new layout may be incomplete. So mdadm takes care to make a backup of those first few stripes and when it assembles an array that was still in the early phase of a reshape it first restores from the backup.
For a reshape that does not change the number of devices, such as changing chunksize or layout, every write will be over-writing the old layout of that same data so after a crash there will definitely be a range of blocks that we cannot know whether they are in the old layout or the new layout or a bit of both. So we need to always have a backup of the range of blocks that are currently being reshaped.
This is the most complex part of the new functionality in mdadm 3.1 (which is not released yet but can be found in the devel-3.1 branch for git://neil.brown.name/mdadm). mdadm monitors the reshape, setting an upper bound of how far it can progress at any time and making sure the area that it is allow to rearrange has writes disabled and has been backed-up.
This means that all the data is copied twice, once to the backup and once to the new layout on the array. This clearly means that such a reshape will go very slowly. But that is the price we have to pay for safety. It is like insurance. You might hate having to pay it, but you would hate it much more if you didn't and found that you needed it.
One way to avoid the extra cost of doing the backup is to increase the number of devices at the same time. e.g. if you had a 4-drive RAID5 you could convert it to a 6-drive RAID6 with a command like mdadm --grow /dev/md0 --level=6 --raid-disk=6 then it will combine the increase in the number of devices with the change to the layout and will copy every block only once (except for the backup it needs to do for the first few stripes).
Reducing the number of devices
The other change worth discussing is the ability to reduce the number of devices. This can be useful to reverse an increase in number of devices that was not intentional, to shrink an array to gain back a spare device, or as part of converting an array from several smaller devices to fewer large devices (thus saving power for example).
While increasing the number of devices or reshaping while leaving the number of devices the same always proceeds from the start of the devices towards the end, reducing the number of devices proceeds from the end of the devices to the beginning. This means that the critical section when a backup is needed happens right at the end of the reshape process, so mdadm runs in the background watching the array and does the backup just when it is needed.
Naturally decreasing the number of devices reduced the amount of available space in the array and the very first write-out to the new arrangement will destroy data that was at the end of the original array. Data that hopefully isn't wanted any more. However people sometimes make mistake and as reducing the number of devices is immediately destructive of data mdadm requires a little more care. In particular, before reducing the number of devices in a RAID5 or RAID6 you must first reduce the size of the array using the new --array-size= option to mdadm --grow. This truncates the array in a non-destructive way. You can check if the data that you care about is still accessible and then when you are sure, use mdadm --grow --raid-disks= to reduce the number of devices.
If you have replaced all the devices with larger devices, you can avoid the need to reduce the size of the array by increasing the component size at the same time as reducing the number of devices. e.g. on a 4-disk RAID5, mdadm --grow --size max --raid-disk 3 ... or at least you should be able to. The current mdadm pre-release don't get that right but hopefully it will before mdadm-3.1 is really released.
Currently supported reshapes
A bit of a summary of what is actually supported seems in order here.RAID1
A RAID1 can change the number of devices or change the size of individual devices. A 2 drive RAID1 can be converted to a 2 drive RAID5.RAID4
A RAID4 can change the number of devices or the size of individual devices. It cannot be converted to RAID5 yet (though that should be trivial to implement).RAID5
A RAID5 can change the number of devices, the size of the individual devices, the chunk size and the layout. A 2 drive RAID5 can be converted to RAID1, and a 3 or more drive RAID5 can be converted to RAID6.RAID6
A RAID6 can change the number of devices, the size of the individual devices, the chunk size and the layout. And RAID6 can be converted to RAID5 by first changing the layout to be similar to RAID5, then changing the level.LINEAR
A lINEAR array can have a device added to it which will simply increase its size.RAID0 and RAID10
These arrays cannot be reshaped at all at present.Future Work
While the work describe here greatly increases the options for reshaping an array there is still more to do (apart from just fixing the bugs in the current code).While conversion from RAID0 to RAID5 was used as an example above, it isn't actually implemented yet. md/RAID0 is broader than the regular definition of RAID0 in that the devices can be different sizes. If they are, then the array cannot be converted to RAID5 at all, so a general conversion is not possible.
It would still be good to support a simple conversion if the RAID0 does just use the same amount of space from each device. It would also be nice to teach RAID5 to handle arrays with devices of different sizes. There are some complications there as you could have a hot spare that can replace some devices but not all. There would also be the need to store the active sizes of all devices in the metadata - something RAID0 doesn't need as it doesn't need to be able to cope with missing devices.
If we did have support for variable drive sizes in a RAID5, then we could implement extending a RAID0 by converting it to a degraded RAID5, perform the conversion there using common code, then convert back to RAID0
It would also be nice to add some reshape options to RAID10. Currently you cannot change a RAID10 at all. The first step here would be enumerating exactly what conversions make sense with the various possible layouts (near/far/offset) and then finding how to implement them most easily. But that is a job for another days.
Comments...
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (19 August 2009, 12:52 UTC) Wow. Impressive work. Thank you for all of your hard and careful work on this.
Minor nit, you have a typo of sunheading rather than subheading in the Raid5 supported operations section.
-ben
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (19 August 2009, 13:31 UTC) Re: Converting RAID5 to RAID6 and other shape changing in md/raid (20 August 2009, 01:02 UTC) I have a question about this new feature.
mdadm --grow --size max --raid-disk 3
Here is what I am doing. I currently have a 6 Drive Raid 5 consisting of 5x1TB, and a 1TB LVM (2x320GB, and 1x500GB (OS included in extra space)). I just recently bought 3 more 1TB drives and was hoping to build a new Raid 5 using 256k chunks instead of 64k chunks. I wanted to gradually remove and shrink the original array freeing up drives and then move the drives over to the new array.
I am getting the error "cannot reduce number of data disks yet", and I am using version 2.6.7.1 in Ubuntu 9.04
Is this possible at all in any version of mdadm, or am I going to have to get creative?
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (20 August 2009, 06:56 UTC) Yes it is possible with the new code. However it isn't quite released yet (Though you can pull it from
my git tree). You will need mdadm-3.1 and linux-2.6.31.
If you just want to change the chunk size, the new mdadm will do that for you
If you want to reduce the number of devices in an array you can. You will have to
shrink the filesystem first, then shrink the array with
and then change the number of devices.
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (20 August 2009, 14:09 UTC) Thanks, I managed to find a way to backup all my data on my old array so I can kill it and just add the drives to the new array.
Hopefully next time I need to shrink it will be released and stable, awesome to see the quick feedback.
I trust mdadm software raid way more than any hardware raid.Allows me to build a relatively cheap system and give me all the protection I need. And it's worked awesome switching between 4 different motherboards now, always detects everything no problem, awesome.
And it's nice to be able to do everything live.
Keep it up, Thanks.
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (21 August 2009, 17:35 UTC) Following your instructions, i upgraded to mdadm, but this command looks not to work
mdadm --grow /dev/md0 --level=6 --raid-disk=6
madam: /dev/md0: Cannot reshape array without increasing size (yet).
I wanted to convert a 5-disk RAID5 to 6-disk RAID6.
If i use this command below, there is another error:
mdadm --grow /dev/md0 --level=6 --raid-disk=6 --size=max
madam: can change at most one of size, raiddisks, bitmap, and layout.
Could you give some instructions? Thanks.
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (22 September 2009, 22:08 UTC) Hi,
I'm considering reshaping an almost full (99%) 7-disk RAID5 array to a 9-disk RAID6 array, but the fact that mdadm-3.1 (which I understand is the only one capable of doing that) hasn't been released (even in -rc) yet has me a little worried...
Do you have any ETA for a release? Do you consider the code stable enough already? I know I should backup before the reshape, but it's not technically (and financially ;-P) feasible; that's why I use el-cheapo software RAID on el-cheapo consumer grade hardware... :)
Thanks for your hard work
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (23 September 2009, 15:37 UTC) No, I don't really have an ETA for a release. Maybe next week, almost certainly before the
end of October. The only thing that I need to do before a release it to write a bunch of
test cases so I can increase my confidence in the code.
I think you should be safe running the code from the 'master' branch of my git tree.
However if anything does go wrong, I suggest that you don't try to fix it, but rather
stop touching the array and email all the details to linux-raid@vger.kernel.org (feel
free to Cc me explicitly). I will work with you to find out exactly what happened and
to fix it. But I really don't expect that to be needed.
Whether it works or not, I would really appreciate an email to linux-raid@vger.kernel.org detailing your experience.
Thanks.
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (07 October 2009, 02:52 UTC) I was wondering how to reduce the number of devices on a RAID-5 without having to buy extra devices for the move... and you just implemented it!
On Linux 2.6.31.2 I performed a test with a RAID-5 of 4 LVM volumes, hosting an ext3 filesystem. After shrinking the filesystem, I reduce the array size with mdadm from the branch devel-3.1 then I reduce the number of devices to 3 and grow the filesystem. It works perfectly.
Thank you and congratulations for all your work.
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (18 October 2009, 20:11 UTC) Dear Neil,
I love this mdadm-stuff, as it adding a huge benefit for linux (and of course for myself). You are talking about future-capabilities, which I would love to see: "If we did have support for variable drive sizes in a RAID5"
My server has actually a raid6 of 6 active 500GB-drives and 1 spare. I would like to add another 1.5TB-drive, but I do not like to split this new drive into 3 * 500GB, as then failing this single 1.5TB drive would render my raid unusable. I have seen ideas of striping e.g. the existing 500GB-disks to a 1-TB-disk, so I could assemble my raid6 with a new 1 *TB native and e.g. 3 * striped 500GB (=3 * 1-TB-disk). However this would result into a be degraded raid6 as well, unless I add 3 *1TB to this raid.
So having the capability of mixing different sizes in a raid6 would be genious. But I guess there would be limitations, as of the size for the spare and I am not sure what would happen if the largest-drive would fail.
However, even with todays flexibility, this mdadm is an outstanding useful tool! congrats!
christian
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (18 October 2009, 23:19 UTC) christian: You can stack the MD levels... So make the 6 500G disks into two 3-disk RAID-0s. Then you have 3 1.5T sized devices (the two RAID-0s and the 1.5T partition) and can build your final RAID-5 on top of that.
Ugly but doable.
-ben
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (19 October 2009, 04:57 UTC) thats what I meant with my (afterwards suboptimal) statement: 3 * striped 500GB. So many thanks, as you suggested what I already thought. I feel now safer going down this road of stacking md-levels. Thanks alot for your useful help
christian
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (21 October 2009, 14:02 UTC) git branch --track devel-3.1 origin/devel-3.1
git checkout -f devel-3.1
make
sudo make install
mdadm - v3.0.2 - 25th September 2009
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (25 December 2009, 00:12 UTC) Firstly -I am impressed with the new RAID6 and reshape features added recently - respect!
After 1-2 days I noted the rate of progress dropped very significantly, and "/proc/mdstat" now reports anything from 5000kbs to 200kbs, mostly around 500kbs. Everything was running OK but just miserably slow. After 4 days its now 71.2% complete.
I have just shutdown and restarted the machine and onreboot it has restarted the reshape and is currently running at a "proper" speed again at between 8000 and 10000 kbs (8-10MB/s). However, it may be after a few hours the rate drops back to 500kbs again?
Is this normal? What have I misconfigured?
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (04 January 2010, 05:04 UTC) Loving the software, have just kicked off a raid 5 to 6 conversion from 6 to 7 drives. It is running at 2384K/sec averagely dunno if that is normal? Used my mirrored raid 1 drive as the backup.
Good job!
(PS it'd be really, really handy if your posts/messages had a date on them?)
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (04 January 2010, 08:44 UTC) Yes, when you do an in-place raid conversion (array size doesn't change) it will always be very slow as all data has to be written twice and there isn't much opportunity for streaming.
And thanks for the suggestion, there should be dates everywhere now.
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (06 January 2010, 07:25 UTC) I was wondering if I could pick your brain and get opinions on my upgrade plan. I am currently running a 4x500GB RAID5 array that is about 80% full now. I've decided to upgrade to 5x2TB drives in a RAID6 configuration for extra redundancy. One thing that is a bit of a problem is that my machine can only support 5 drives, so I can't just attach the new drives and copy the data. I've come up with two options to do this upgrade:
1. Add one 2TB drive as a stand-alone, copy all data to the 2TB drive, replace the 4 old drives, create 4x2TB RAID6 array, copy data over from stand-alone drive, and finally expand the array to include the fifth drive.
2. Add 2TB drive, make current RAID5 array a RAID6 array including that drive, then swap out the 4 500GB drives one at a time as if they had each died.
I'm leading towards option 1 as I think that would be quicker, the longest part being the expanding of the array from 4 drives to 5 at the end, but the copying to and from the 2TB drive will probably just take several hours each way. It also seems fairly safe because I'll have the 4 500GB unaltered if anything should go wrong.
Thanks for any input, and thanks for all your hard work on this project.
Keith
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (07 January 2010, 06:34 UTC) Option 1 is definitely better if you are happy with the extended down-time that it requires. It will very likely be faster as the RAID5 -> RAID6 conversion is very slow.
As you say, you will have a complete, valid copy of the data on the smaller drives which is good insurance.
So go with option 1.
NeilBrown
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (08 January 2010, 04:33 UTC) Keith
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (09 January 2010, 17:43 UTC) Neil was wondering if I could get your advice. I'm currently running at 4x1TB Raid5 and I would like to change that to a 6x1TB Raid6. I guess confused on the process. So essentially i can just use the mdadm --add /dev/mdX /dev/sdX and add the 2 drives making it 6x1TB. Then mdadm --grow /dev/md0 --level=6 --raid-disk=6 will correctly reshape my raid5 to a raid6 then i'd just have to xfs_growfs /dev/mdX afterwards? Just wondering if my thought process is correct here. Thanks in advance this is some great progress.
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (10 January 2010, 05:11 UTC) Hi all, I'm converting a 7 disk RAID-5 array to an 8 disk RAID-6 array and wanted to share. Unlike the previous poster, I am not increasing the size of the RAID device, I am simply increasing redundancy.
First of all I made sure I was running mdadm 3.1 or above, which my distro (gentoo) did not have by default. So I installed mdadm 3.1.1, which I believe is current as of the time of this writing.
To add the 8th disk to my existing RAID-5 array, I ran:
mdadm --add /dev/md3 /dev/sdh3
this added /dev/sdh3 as a hot spare. Then, to convert this to a RAID-6 array, I ran:
mdadm --grow /dev/md3 --level=6 --raid-devices=8 --backup-file=/nfs/media/tmp/md3.backup
Notice the argument is "--raid-devices", not "--raid-disk" as in Neil's post.
I had tried to run the --grow command without the --backup-file argument, as Neil's post seems to say that a backup file is not necessary when a hot spare is present. But mdadm wasn't having it, it told me:
mdadm level of /dev/md3 changed to raid6
mdadm: /dev/md3: Cannot grow - need backup-file
mdadm: aborting level change
With the --backup-file argument everything seems to be working fine. Here's the relevant part of my /proc/mdstat:
md3 : active raid6 sdh3[7] sdg3[6] sdf3[5] sde3[4] sda3[0] sdb3[2] sdc3[3] sdd3[1]
120052224 blocks super 0.91 level 6, 256k chunk, algorithm 18 [8/7] [uuuuuuu_]
[=>...................] reshape = 6.3% (1269760/20008704) finish=151.2min speed=2064K/sec
My next step is to convert my 4 terabyte /dev/md5 to a RAID-6 array. Neil wasn't kidding when he said the reshape is a slow process... at the rate that /dev/md3 is converting, I estimate that it will take 4.5 days to convert my /dev/md5 to RAID-6.
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (12 January 2010, 23:49 UTC) mdadm 3.1.1
kernel 2.6.32.10
/dev/md1 is a 3 disk raid5 64k chunk size.
/mnt/tmp is a spare hard drive.
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (25 January 2010, 09:47 UTC) Re: Converting RAID5 to RAID6 and other shape changing in md/raid (12 February 2010, 22:16 UTC) I have just started a conversion of a 6 disk 2TB RAID5 to a 8 disk 2TB RAID6 on my system.
The performance during conversion was terrible - 1MB/second. This got me suspicious, as I know the array could perform much better than that.
Issuing "echo 200000 > /sys/block/<mddevice>/md/sync_speed_min" fixed it, as it now reshapes at 20MB/second.
This ofcourse blocks everything else, but issuing an "echo 10000" gives the a fair tradeoff, as the "when idle" system obviously seems to have a problem.
So try this, instead of rebooting.
Comment (22 February 2010, 12:03 UTC) first of all I want to thank Neil and all other developers of mdadm for their great work. I have been using mdadm for several years now and my arrays have survived two disk-faults flawlessly - from the notification-email to rebuilding the array everything went as it should.
I have a question regarding my raid6-array, consisting of 7 1TB-disks providing a single lvm-physical volume. I plan to expand that array with a 1.5TB-disk and want to make sure that I got everything written here and on the mailing list right:
- If I grow that array after adding the new 1.5TB-disk to it, only 1TB of the new disk will be used and the array-size will increase by 1TB. (Do I have to issue some special commands when growing or will mdadm automatically just use 1TB of the 1.5TB-disk?)
- If I resize the lvm-physical volume the array provides, the 1TB will become usable by lvm.
Is that correct?
Then I got another question: To grow a raid6-array, I need mdadm past version 3.x. Currently, the array is on a debian-server running a year or so without reboot, therefore it runs on mdadm-2.6.x. I recently updated the mdadm-package to the current version 3.1.1 using aptitude.
Is a reboot necessary for the grow-feature to become available or does it suffice to just stop and start the array (or is neither of these actions necessary)?
Thank you all for your help!
rman
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (22 February 2010, 11:03 UTC) - yes, only 1TB of the new device will be used. md doesn't need to be told any special for this to happen.
- similarly, lvm will only be able to see 1TB of extra data.
Providing your kernel is v2.6.21 or more recent you should not need to reboot. If your kernel is older than
that, you will need a new kernel.
Comment (23 February 2010, 03:34 UTC) md11 : active raid6 sdh[9](S) sdc[0] sdi[8] sdf[7] sde[4] sdk[3] sdj[2] sdd[1]
4883811840 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/7] [uuuuuuu]
If i now issue 'mdadm /dev/md11 --grow --raid-devices=8 --backup-file=/root/backup_md11.bak' I get
mdadm: this change will reduce the size of the array.
use --grow --array-size first to truncate array.
e.g. mdadm --grow /dev/md11 --array-size 1565606912
Why would adding a disk REDUCE the size? Am I missing something here? If I add the '--size=max' switch to the command line, i.e.
'mdadm --grow /dev/md11 --raid-devices=8 --level=6 --backup-file=/root/backup_md11.bak --size=max', I get
mdadm: cannot change component size at the same time as other changes.
Change size first, then check data is intact before making other changes.
I really don't understand this as the new disk is exactly the same model as the other ones. 'hdparm -I' on the new disk gives me a device size which is identical to the other disks. Has anyone a clue what is going on? Has it something to do with the non-standard chunk-size?
Thanks for helping,
rman
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (23 February 2010, 07:56 UTC) You have found a bug! It has been fixed for a while, but I haven't released an updated version yet.
The two patches to fix it are:
http://neil.brown.name/git?p=mdadm;a=commitdiff;h=2ed4f75388f99968be58097941a9704f6e42d701
and
http://neil.brown.name/git?p=mdadm;a=commitdiff;h=f98841b3852ceb7fce56a6f818236a4af9b5a00a
If you have 'git' installed, you might do best to:
git clone git://neil.brown.name/mdadm mdadm
cd mdadm
make install
and try that.
Re: Converting RAID5 to RAID6 and other shape changing in md/raid (06 March 2010, 08:41 UTC) has changing the chunk size been fixed?
i have a 3 disk raid5 that has 64k chunks and i want to go to 256k chunks.
trying to do this crashed the array and forced a resync
Re: Changing chunk size (06 March 2010, 09:41 UTC) As far as I know, changing chunksize works OK.
There might be some issues when going from a smaller to a larger chunksize if the size of the components is not a multiple of the larger chunk size - mdadm doesn't seem to check that properly at the moment, but the failure mode is to do nothing, not to crash the array.
If you have specific details - kernel log error messages, "mdadm --detail" details of the array, version of kernel and mdadm, and anything else that might help me reproduce your problem, please post them to linux-raid@vger.kernel.org (you don't have to subscribe to post).
I just successfully reshaped a small scratch 3-drive-raid5 from 64K chunks to 256K chunks using mdadm-3.1.1 and linux-2.6.32.
I've fixed the sunheading - thanks.
mdadm --grow /dev/mdX --chunk-size=128
mdadm --grow --array-size
Neil,
Dear Ben,
So I guess I'm the newbie here, because I can't seem to get mdadm v3.1 installed...
git clone git://neil.brown.name/mdadm
All seems to work just fine, until mdadm --version:
Even doing ./mdadm --version within the devel-3.1 branch shows 3.0.2
Am I just confused, or did I do something wrong.
I am currently reshaping a RAID6 array from 6x1.5TB discs to 8x1.5TB discs. Simply - I have added 2 identical discs to an existing (almost empty) 6disc x RAID6 array. The operation started a 4 days ago and reporting running at 7000kb/s, and was consuming 5-6% CPU (load average 1.4-1.6 from memory). This was slower than I expected. (No raid controller, using 5x onboard SATA ports and 2x 2-port SIL SATA PCE-Express cards, Quad Core AMD 3GHz, 4GB RAM etc)
Neil,
Neil, thanks for the input. I'll go that route and I don't think the downtime should be too bad as the array is primarily media storage. I won't be able to write to it for a while, but I'll probably be able to stream from it without much trouble.
sudo mdadm --grow /dev/md1 --chunk=128 --backup-file=/mnt/tmp/backup
locks up the computer and forces a resync.
#####
mdX : active raid5 sdd1[8](S) sdb1[7](S) sdf8[0] sdl8[4] sdk2[5] sdc1[6] sdj6[3] sdi8[1]
. Y blocks super 1.1 level 5, 128k chunk, algorithm 2 [6/6] [uuuuuu]
# mdadm --grow /dev/mdX --level=6 --raid-devices=8 --backup-file=/root/mdX.backupfile
mdX : active raid6 sdd1[8] sdb1[7] sdf8[0] sdl8[4] sdk2[5] sdc1[6] sdj6[3] sdi8[1]
. Y blocks super 1.1 level 6, 128k chunk, algorithm 18 [8/9] [uuuuuu_u]
. [>....................] reshape = 0.0% (33920/484971520) finish=952.6min speed=8480K/sec
Nooo mdadm 3.1.1 I wanted an 8 device raid-6 Why do you show 9?
-- update
What is it showing me now??? blocks super 1.1 level 6, 128k chunk, algorithm 2 [8/10] [uuuuuuuu]
mdadm --detail /dev/mdX
/dev/mdX:
. Version : 1.01
. Raid Level : raid6
. Array Size : 6z ()
. Used Dev Size : z ()
. Raid Devices : 8
. Total Devices : 8
. Persistence : Superblock is persistent
. Update Time : Sun Jan 24 14:40:32 2010
. State : clean
.Active Devices : 8
Working Devices : 8
.Failed Devices : 0
. Spare Devices : 0
. Layout : left-symmetric
. Chunk Size : 128K
. Number Major Minor RaidDevice State
. 0 8 . 0 active sync /dev/sdf
. 1 8 . 1 active sync /dev/sdi
. 3 8 . 2 active sync /dev/sdj
. 6 8 . 3 active sync /dev/sdc
. 5 8 . 4 active sync /dev/sdk
. 4 8 . 5 active sync /dev/sdl
. 8 8 . 6 active sync /dev/sdd
. 7 8 . 7 active sync /dev/sdb
... Did it actually do what I want but just show me the wrong result with kernel 2.6.32-gentoo-r2
Hey there,
Thank you very much for your answers, everything as I expected. I decided to add another 1TB disk instead of a 1.5TB disk as I didn't consider that the 1.5TB disk would be much slower than all other preexisting 1TB disk.
So here my situation again: I want to add another 1TB-disk to an existing raid6-array consisting of seven 1TB-disks. Unfortunately mdadm gives me some strange errors.
At first I added the new disk (sdh) to the existing array (md11), which went fine. So now I have a 7-disk array with a single spare: