[PLUG] Unable to reactivate a RAID10 mdadm device
Arun Khan
knura9 at gmail.com
Tue Feb 12 13:53:12 IST 2013
Recovery OS -- System Rescue CD v 2.8.0
Production OS - Debian Squeeze (6) 2.6.32 stock kernel, using mdadm raid
/dev/md0 in raid level RAID10 - members /dev/sdb1, /dev/sdc1,
/dev/sdd1, /dev/sde1 all with partion id=fd
HDD /dev/sdb went bad, replaced it with another disk with same size
partion (id=fd)
using System Rescue CD v2.8.0
1. System Rescue CD recognized the md devices but it comes up as 'inactive'
Searched for possible solutions and I have tried several things including
zeroing the super block and adding them back to the array.
Still unable to bring back /dev/md0 with all 4 partions in active mode.
I have included below, the entire transcript of the commands I have
tried to recover /dev/md0
I have data on /dev/md0 that I need. I do have back ups of critical
files (but not all).
I prefer solving the problem v/s recreating the /dev/md0 from scratch.
Any help in solving this problem would be highly appreciated.
TIA,
-- Arun Khan
---- transcript of mdadm activity with System Rescue CD v2.8.0 ----
# mdadm -V
mdadm - v3.1.4 - 31st August 2010
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md0 : inactive sdd1[2] sde1[3]
312574512 blocks super 1.0
# mdadm -S /dev/md0
mdadm: stopped /dev/md0
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
unused devices: <none>
# mdadm -v -v -A /dev/md0 -R /dev/sd[bcde]1
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 3.
mdadm: added /dev/sdb1 to /dev/md0 as 0
mdadm: added /dev/sdc1 to /dev/md0 as 1
mdadm: added /dev/sde1 to /dev/md0 as 3
mdadm: added /dev/sdd1 to /dev/md0 as 2
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error
mdadm: Not enough devices to start the array.
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md0 : inactive sdd1[2] sde1[3]
312574512 blocks super 1.0
unused devices: <none>
from /var/log/messages
Feb 12 06:13:59 sysresccd kernel: [ 7593.339015] md: md0 stopped.
Feb 12 06:13:59 sysresccd kernel: [ 7593.374016] md: bind<sdb1>
Feb 12 06:13:59 sysresccd kernel: [ 7593.374417] md: bind<sdc1>
Feb 12 06:13:59 sysresccd kernel: [ 7593.374604] md: bind<sde1>
Feb 12 06:13:59 sysresccd kernel: [ 7593.374869] md: bind<sdd1>
Feb 12 06:13:59 sysresccd kernel: [ 7593.374899] md: kicking non-fresh
sdc1 from array!
Feb 12 06:13:59 sysresccd kernel: [ 7593.374903] md: unbind<sdc1>
Feb 12 06:13:59 sysresccd kernel: [ 7593.379016] md: export_rdev(sdc1)
Feb 12 06:13:59 sysresccd kernel: [ 7593.379041] md: kicking non-fresh
sdb1 from array!
Feb 12 06:13:59 sysresccd kernel: [ 7593.379044] md: unbind<sdb1>
Feb 12 06:13:59 sysresccd kernel: [ 7593.386010] md: export_rdev(sdb1)
Feb 12 06:13:59 sysresccd kernel: [ 7593.387382] md/raid10:md0: not
enough operational mirrors.
Feb 12 06:13:59 sysresccd kernel: [ 7593.387410] md: pers->run() failed ...
# mdadm -D /dev/md0
/dev/md0:
Version : 1.0
Creation Time : Fri Apr 29 04:27:04 2011
Raid Level : raid10
Used Dev Size : 156287232 (149.05 GiB 160.04 GB)
Raid Devices : 4
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Mon Feb 11 13:43:52 2013
State : active, FAILED, Not Started
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Layout : near=2
Chunk Size : 32K
Name : brahmaputra:0
UUID : f2d4e898:2e026f85:244a7e9c:908e1af7
Events : 783527
Number Major Minor RaidDevice State
0 0 0 0 removed
1 0 0 1 removed
2 8 49 2 active sync /dev/sdd1
3 8 65 3 active sync /dev/sde1
# mdadm -E /dev/sdc1
/dev/sdc1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : f2d4e898:2e026f85:244a7e9c:908e1af7
Name : brahmaputra:0
Creation Time : Fri Apr 29 04:27:04 2011
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 312574512 (149.05 GiB 160.04 GB)
Array Size : 625148928 (298.09 GiB 320.08 GB)
Used Dev Size : 312574464 (149.05 GiB 160.04 GB)
Super Offset : 312574640 sectors
State : clean
Device UUID : 2fbc103e:ca40e0c2:b8e4d64f:0fbc7b94
Internal Bitmap : -8 sectors from superblock
Update Time : Mon Feb 11 13:43:52 2013
Checksum : 2e8e9fad - correct
Events : 0
Layout : near=2
Chunk Size : 32K
Device Role : spare
Array State : ..AA ('A' == active, '.' == missing)
# mdadm -E /dev/sdb1
/dev/sdb1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : f2d4e898:2e026f85:244a7e9c:908e1af7
Name : brahmaputra:0
Creation Time : Fri Apr 29 04:27:04 2011
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 312574512 (149.05 GiB 160.04 GB)
Array Size : 625148928 (298.09 GiB 320.08 GB)
Used Dev Size : 312574464 (149.05 GiB 160.04 GB)
Super Offset : 312574640 sectors
State : clean
Device UUID : e27c187b:9004cb93:5bb05639:164822cd
Internal Bitmap : -8 sectors from superblock
Update Time : Mon Feb 11 13:43:52 2013
Checksum : 5ea77bd0 - correct
Events : 0
Layout : near=2
Chunk Size : 32K
Device Role : spare
Array State : ..AA ('A' == active, '.' == missing)
# mdadm -E /dev/sde1
/dev/sde1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : f2d4e898:2e026f85:244a7e9c:908e1af7
Name : brahmaputra:0
Creation Time : Fri Apr 29 04:27:04 2011
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 312574512 (149.05 GiB 160.04 GB)
Array Size : 625148928 (298.09 GiB 320.08 GB)
Used Dev Size : 312574464 (149.05 GiB 160.04 GB)
Super Offset : 312574640 sectors
State : clean
Device UUID : 78fc82bc:2eb18f07:56c98922:7639269e
Internal Bitmap : -8 sectors from superblock
Update Time : Mon Feb 11 13:43:52 2013
Checksum : ce19a703 - correct
Events : 783527
Layout : near=2
Chunk Size : 32K
Device Role : Active device 3
Array State : ..AA ('A' == active, '.' == missing)
# mdadm -E /dev/sdd1
/dev/sdd1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : f2d4e898:2e026f85:244a7e9c:908e1af7
Name : brahmaputra:0
Creation Time : Fri Apr 29 04:27:04 2011
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 312574512 (149.05 GiB 160.04 GB)
Array Size : 625148928 (298.09 GiB 320.08 GB)
Used Dev Size : 312574464 (149.05 GiB 160.04 GB)
Super Offset : 312574640 sectors
State : clean
Device UUID : 1dce2f63:fef488cb:1a362c57:7ed908ac
Internal Bitmap : -8 sectors from superblock
Update Time : Mon Feb 11 13:43:52 2013
Checksum : 7b44c944 - correct
Events : 783527
Layout : near=2
Chunk Size : 32K
Device Role : Active device 2
Array State : ..AA ('A' == active, '.' == missing)
# mdadm --zero-superblock /dev/sdb1
# mdadm -E /dev/sdb1
mdadm: No md superblock detected on /dev/sdb1.
# mdadm /dev/md0 --add /dev/sdb1
mdadm: add new device failed for /dev/sdb1 as 4: Invalid argument
from /var/log/messages
Feb 12 06:52:26 sysresccd kernel: [ 9900.298880] md0: ADD_NEW_DISK not supported
More information about the Plug-mail
mailing list