Hot-swapping the hard drives when the RAID crashes

«No server downtime when you need to replace the RAID drives»

Contents

• Logical volume status when the RAID operates normally
• When a drive fails, follow the steps below to check the drive status:
• Install a new drive to rebuild RAID 5 by hot swapping

Procedure of hot-swapping the hard drives when the RAID crashes

RAID 5 disk mirroring provides highly secure data protection. You can use two hard drives of the same capacity to create a RAID 5 array. The RAID 5 creates an exact copy of data on the member drives. RAID 5 protects data against single drive failure. The usable capacity of RAID 5 is the capacity of the smallest member drive. It is particularly suitable for personal or company use for important data saving.

Logical volume status when the RAID operates normally

When the RAID volume operates normally, the volume status is shown as Ready under ‘Disk Management’ > ‘Volume Management’ section.

When a drive fails, follow the steps below to check the drive status:

When RAID volume operates normally, the volume status is shown as Ready in the Current Disk Volume Configuration section.

• The server beeps for 1.5 sec twice when the drive fails.
• The Status LED flashes red continuously.
• Check the Current Disk Volume Configuration section. The volume status is In degraded mode.

You can check the error and warning messages for drive failure and disk volume in degraded mode respectively in the Event Logs.

Note: You can send and receive alert e-mail by configuring the alert notification. For the settings, please refer to the System Settings/ Alert Notification section in the user manual.

Install a new drive to rebuild RAID 5 by hot swapping

Please follow the steps below to hot swap the failed hard drive:

• Prepare a new hard drive to rebuild RAID 5. The capacity of the new drive should be at least the same as that of the failed drive.
• Insert the drive to the drive slot of the server. The server beeps for 1.5 seconds twice. The Status LED flashes red and green alternatively.
• Check the Current Disk Volume Configuration section. The volume status is Rebuilding and the progress is shown.

• When the rebuilding is completed, the Status LED lights in green and the volume status is Ready. RAID 5 mirroring protection is now active.
• You can check the disk volume information in the Event Logs.

Note: Do not install the new drive when the system is not yet in degraded mode to avoid unexpected system failure.

Obtained from this link

QNAP RAID System Errors & How To Fix

I – Introduction;

II -How to Fix if RAID seems “In Degreed”

III -How to Fix if RAID seems “In Degreed Mode, Read Only, Failed Drive(s) x” Cases;

IV – How to Fix if RAID Becomes “Unmounted” or “Not Active”

V – “Recover” Doesnt Work, How to Fix if RAID Becomes “Unmounted” or “Not Active”

VI – Stuck at Booting / Starting Sevices, Please Wait / Cant Even Login Qnap Interface

VII – IF “config_util 1″ Command gives “Mirror of Root Failed” Eror;

VIII – How to Fix “Broken RAID” Scenario

IX – If None of These Guides Works;

..

I – Introduction;

Warning : This documents are recomended for Professional users only. If you dont know what you’r doing, you may damage your RAID which cause loosing data. Qnapsupport Taiwan works great to solve this kind of RAID corruptions easly, and My advice is directly contact with them at this kind of cases.

1 – If documetn says “Plug out HDD and Plug in” always use original RAID HDD at the same slot. Dont use new HDDs!

2 – If you can access your data after these process, backup them quickly.

3 – You can loose data on Hardware RAID devices and other NAS brands, but nearly you cant loose data on Qnap, but recovering your datas may cause time lost, so Always double backup recomended for future cases.

4 – If one of your HDD’s cause Qnap to restart itself when you plug in to NAS, Dont use that HDD to solve these kind of cases.

This document is not valid for Qnap Ts-109 / Ts-209 / Ts-409 & Sub Models.

..

II -How to Fix if RAID seems “In Degreed”

If your RAID system seems as down below, use this document. If not, Please dont try anything in this document:

In this case, RAID information seems “RAID 5 Drive 1 3″ so, your 2.th HDD is out of RAID. Just plug out broken HDD from 2.Th HDD slot, wait around 15 seconds, and Plug in new HDD with the same size.

..

III -How to Fix if RAID seems “In Degreed Mode, Read Only, Failed Drive(s) x” Cases;

If your system seems In Degraded, Failed Drive X, you probably loose more HDD than RAID tolerated, so;

1 – Take your Backup,

2 – Re-Install Qnap From Begining.

Qnap data protection features doesnt let you loose data even if your 5 HDD gives Bad sector errors in 6 HDD RAID Systems.

..

IV – How to Fix if RAID Becomes “Unmounted” or “Not Active”

If your RAID system seems as the picture down below, fallow this document. If not, Please dont try anything in this document:

1 – Update your Qnap fimware with Qnapfinder 3.7.2 or higher firmware. Just go to Disk Managment -> RAID managment. Choose your RAID and press “Recover” to fix.

If this doesnt work and “Recover” button is still avaible, just fullow these steps;

1 – While device is still working, Plug out HDD that you suspect which maybe broken, and press Recover button again.

2 – If you plug out HDD —> then Plug it in the same slot again —> Press “Recover” Once Again and it should repair RAID;

3 – Here is log files;

4 – And RAID systems comes back with “Indegreed Mode”, so just quickly backup your datas, and Reinstall Qnap without broken HDD again.

We loose 2 HDD from RAID 5, so its impossable to fix this RAID again. Just Re-install.

..

V – “Recover” Doesnt Work, How to Fix if RAID Becomes “Unmounted” or “Not Active”

1 – Download Putty & login Qnap,

2 – Make sure the raid status is active

To understant, type :

#more /proc/mdstat

Also If you want to Stop Running Services;

# /etc/init.d/services.sh stop

Unmount the volume:

# umount /dev/md0

Stop the array:

# mdadm -S /dev/md0

Now Try This command

For RAID 5 with 4 HDD’s

mdadm -CfR –assume-clean /dev/md0 -l 5 -n 4 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3

-l 5 : means RAID 5. If its RAID 6, try -l 6;

-n 4 means number of your HDDs, If you have 8 HDD, try –n 8

/dev/sda3 means your first HDD.

/dev/sdd3 means your 4.th HDD

Example;

If you have RAID 6 with 8 HDD, change command line with this;

mdadm -CfR –assume-clean /dev/md0 -l 6 -n 8 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3 /dev/sde3 /dev/sdf3 /dev/sdg3 /dev/sdh3

If one of your HDD has Hardware Error, which cause Qnap restart itself, plug out that HDD, and type “missing” command for that HDD;

Example, if your 2.th HDD is broken, use this command;

# mdadm -CfR –assume-clean /dev/md0 -l 5 -n 4 /dev/sda3 missing /dev/sdc3 /dev/sdd3

3 – try manually mount

# mount /dev/md0 /share/MD0_DATA -t ext3

# mount /dev/md0 /share/MD0_DATA -t ext4

# mount /dev/md0 /share/MD0_DATA -o ro (read only)

4 – So, Result Should be Like That;

..

VI – Stuck at Booting / Starting Sevices, Please Wait / Cant Even Login Qnap Interface

Just Plug out broken HDD and restart Qnap again. This should restarts Qnap back again.

If you’r not sure which HDD is broken, Please follw this steps;

1 – Power Off the NAS.

2 – Plug out All HDDs,

3 – Start Qnap without HDDs,

4 – Qnapfinder Should find Qnap in a few minutes. Now, Plug in all HDD’s back again same slots.

5 – Download Putty from this link and login with admin / admin usernam / password;

http://www.chiark.greenend.org.uk/~sgta … nload.html

6 – Type this command lines which I marked blue;

# config_util 1 -> If result of this command give “Root Failed” dont go on and contact with Qnapsupport team

# storage_boot_init 1 ->

# df -> IF dev/md9 (HDA_ROOT) seems full, please contact with Qnapsupport.

(Now, you can Reboot your device with this command. ıf you want to reset your configration, please skip this.)

# reboot

..

VII – IF “config_util 1″ Command gives “Mirror of Root Failed” Eror;

1 – Power Off the NAS.

2 – Plug out All HDDs,

3 – Start Qnap without HDDs,

4 – Qnapfinder Should find Qnap in a few minutes. Now, Plug in all HDD’s back again same slots.

5 – Download Putty from this link and login with admin / admin usernam / password;

http://www.chiark.greenend.org.uk/~sgta … nload.html

6 – Type this command lines which I marked blue;

# storage_boot_init 2 -> (this time type storage_boot_init 2, not storage_boot_init 1)

This command should turn Qnap back to last “Indegreed” Mode and you should get this kind of information after this command:

7 – Download Winscp and Login Qnap. Go to ->share ->MD0_DATA folder, and backup your datas Quickly.

..

VIII – How to Fix “Broken RAID” Scenario

First, I try to fix “Recovery” method, but doesnt work. At Qnap RAID managment menu, I check All HDDs, But all of them seems good. So I Login with Putty, and type these commands;

mdadm -E /dev/sda3

mdadm -E /dev/sdb3

mdadm -E /dev/sdc3

mdadm -E /dev/sdd3

Except first HDD, other 3 HDD’s doesn have md superblock; Also I try “config_util 1” & “storage_boot_init 2” commands, but both of them gives error;

Costumer got RAID 5 (-l 5) with 4 HDD (-n 4), so ı type this command;

# mdadm -CfR –assume-clean /dev/md0 -l 5 -n 4 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3

then mount with this command;

# mount /dev/md0 /share/MD0_DATA -t ext4

And works perfect.

Also here is Putty Steps;

login as: admin

admin@192.168.101.16′s password:

[~] # mdadm -E /dev/sda3

/dev/sda3:

Magic : a92b4efc

Version : 00.90.00

UUID : 2d2ee77d:045a6e0f:438d81dd:575c1ff3

Creation Time : Wed Jun 6 20:11:14 2012

Raid Level : raid5

Used Dev Size : 1951945600 (1861.52 GiB 1998.79 GB)

Array Size : 5855836800 (5584.56 GiB 5996.38 GB)

Raid Devices : 4

Total Devices : 4

Preferred Minor : 0

Update Time : Fri Jan 11 10:24:40 2013

State : clean

Active Devices : 4

Working Devices : 4

Failed Devices : 0

Spare Devices : 0

Checksum : 8b330731 – correct

Events : 0.4065365

Layout : left-symmetric

Chunk Size : 64K

Number Major Minor RaidDevice State

this 0 8 3 0 active sync /dev/sda3

0 0 8 3 0 active sync /dev/sda3

1 1 8 19 1 active sync /dev/sdb3

2 2 8 35 2 active sync /dev/sdc3

3 3 8 51 3 active sync /dev/sdd3

[~] # mdadm -E /dev/sdb3

mdadm: No md superblock detected on /dev/sdb3.

[~] # mdadm -CfR –assume-clean /dev/md0 -l 5 -n 4 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3

mdadm: /dev/sda3 appears to contain an ext2fs file system

size=1560869504K mtime=Fri Jan 11 10:22:54 2013

mdadm: /dev/sda3 appears to be part of a raid array:

level=raid5 devices=4 ctime=Wed Jun 6 20:11:14 2012

mdadm: /dev/sdd3 appears to contain an ext2fs file system

size=1292434048K mtime=Fri Jan 11 10:22:54 2013

mdadm: array /dev/md0 started.

[~] # more /proc/mdstat

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]

md0 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[0]

5855836800 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md4 : active raid1 sda2[2](S) sdd2[0] sdc2[3](S) sdb2[1]

530048 blocks [2/2] [UU]

md13 : active raid1 sda4[0] sdc4[3] sdd4[2] sdb4[1]

458880 blocks [4/4] [UUUU]

bitmap: 0/57 pages [0KB], 4KB chunk

md9 : active raid1 sda1[0] sdc1[3] sdd1[2] sdb1[1]

530048 blocks [4/4] [UUUU]

bitmap: 1/65 pages [4KB], 4KB chunk

unused devices:

[~] # mount /dev/md0 /share/MD0_DATA -t ext4

[~] #

Here is Result;

..

IX – If None of These Guides Works;

For rarely, I cant solve these kind of problems on Qnap. and sometimes I plug HDD’s to another Qnap to solve these kind of problems.

User different Qnap which has different hardware to solve this kind of cases!;

When you plug HDD, and restart device, Qnap will ask you to to Migrate or Reinstall Qnap, choose “Migrate” Option.

In this case, Qnap coulnt boot at Ts-659 Pro, putty commands doesnt work and we cant even enter Qnap admin interface;

HDD’s couldnt boot from Ts-659 Pro, but after installing different device (At this case, I plug them to Ts-509 Pro) device boots fine. But of course you must use 6 or more HDD supported Qnap to save your datas.

So, If evertything fails, Just try these process with another Qnap.

Obtained from this link

Stuct At Booting When HDD’s Are Not Plugged In

If you cannot access the NAS after Step 3, please do the following:

1. Turn off the NAS.
2. Take out all the hard disk drives.
3. Restart the NAS.

You will hear a beep after pressing the power button, followed by 2 beeps 2 minutes later. If you cannot hear first beep, Please contact your local reseller or distributor for repair or replacement service.

If you cannot Hear the two beeps, and Qnapfinder couldnt find your NAS, the NAS Firmware is Damaged. To fix this problem, please follow “Qnap firmware Recovery / Reflash” Documents for your device model.

If you couldnt solve problem by yourself, Please contact your local reseller or distributor for repair or replacement service

If Qnapfinder can find Qnap, fallow these steps;

1 – Download Putty software;

http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

2 – Plug in all of your HDD’s with right order while device is still working. Dont restart Qnap yet. Check if all HDD’s are allright and recognized by Qnap. If any of HDD doesnt recovnized or size seems “0″, plug out that HDD.

3 –Log with putty by entering the Qnap IP / user name / password. (Username / Password: admin / admin. Port need to enter 22.)

Now enter these command down below; (Choose command from this screen and “copy” Then go to putty, just pr “pess right mouse button once. By this way, you can paste commands automaticly)

# config_util 1 -> (it must say “mirror of root succeed”. if it gives “mirror of root failed” error, stop this step and request help from Qnapsupport.)

# storage_boot_init 1

# df

If dev/md9 (HDA_ROOT) appears full, please contact QNAP support team

# reboot

Now Qnap should reboot well. If you can reach Qnap interface after restart, check RAID system, and change broken HDD with a new one.

Obtained from this link

QNAP TS-212 How to rebuild RAID manually from telnet

Scenario = replace a disk in QNAP TS-212 with RAID 1 configuration active

RAID rebuild should start automatically, but some times it could happen you got stuck with 1 Single Disk + 1 Mirroring Disk Volume:

According to the QNAP Support – How can I migrate from Single Disk to RAID 0/1 in TS-210/TS-212? , TS-210/TS-212 does not support Online RAID Level Migration. Therefore, please backup the data on the single disk to another location, install the second hard drive, and then recreate the new RAID 0/1 array (hard drive must be formatted).

The workaround is this:

1. Telnet to NAS as Admin
2. Check your current disk configuration for Disk #1 and Disk #2 =
   fdisk -l /dev/sda
   Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
   255 heads, 63 sectors/track, 121601 cylinders
   Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot Start End Blocks Id System
   /dev/sdb1 1 66 530125 83 Linux
   /dev/sdb2 67 132 530142 83 Linux
   /dev/sdb3 133 121538 975193693 83 Linux
   /dev/sdb4 121539 121600 498012 83 Linux

   fdisk -l /dev/sdb

   Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
   255 heads, 63 sectors/track, 121601 cylinders
   Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot Start End Blocks Id System
   /dev/sda1 1 66 530125 83 Linux
   /dev/sda2 67 132 530142 83 Linux
   /dev/sda3 133 121538 975193693 83 Linux
   /dev/sda4 121539 121600 498012 83 Linux

3. SDA is the first disk, SDB is the second disk
4. Verify the current status of RAID with this command =
   mdadm –detail /dev/md0
   /dev/md0:
   Version : 00.90.03
   Creation Time : Thu Sep 22 21:50:34 2011
   Raid Level : raid1
   Array Size : 486817600 (464.27 GiB 498.50 GB)
   Used Dev Size : 486817600 (464.27 GiB 498.50 GB)
   Raid Devices : 2
   Total Devices : 1
   Preferred Minor : 0
   Persistence : Superblock is persistent

   Intent Bitmap : Internal

   Update Time : Thu Jul 19 01:13:58 2012
   State : active, degraded
   Active Devices : 1
   Working Devices : 1
   Failed Devices : 0
   Spare Devices : 0

   UUID : 72cc06ac:570e3bf8:427adef1:e13f1b03
   Events : 0.1879365

   Number Major Minor RaidDevice State
   0 0 0 0 removed
   1 8 3 1 active sync /dev/sda3

5. As you can see the /dev/sda3 is working, so disk #1 is OK, but disk #2 is missing from RAID
6. Check if Disk #2 /dev/sdb is mounted (it should be) =
   mount/proc on /proc type proc (rw)
   none on /dev/pts type devpts (rw,gid=5,mode=620)
   sysfs on /sys type sysfs (rw)
   tmpfs on /tmp type tmpfs (rw,size=32M)
   none on /proc/bus/usb type usbfs (rw)
   /dev/sda4 on /mnt/ext type ext3 (rw)
   /dev/md9 on /mnt/HDA_ROOT type ext3 (rw)
   /dev/md0 on /share/MD0_DATA type ext4 (rw,usrjquota=aquota.user,jqfmt=vfsv0,user_xattr,data=ordered,delalloc,noacl)
   tmpfs on /var/syslog_maildir type tmpfs (rw,size=8M)
   /dev/sdt1 on /share/external/sdt1 type ufsd (rw,iocharset=utf8,dmask=0000,fmask=0111,force)
   tmpfs on /.eaccelerator.tmp type tmpfs (rw,size=32M)
   /dev/sdb3 on /share/HDB_DATA type ext3 (rw,usrjquota=aquota.user,jqfmt=vfsv0,user_xattr,data=ordered,noacl)

   
   

    

7. Dismount the /dev/sdb3 Disk #2 with this command =
   umount /dev/sdb3
8. Add Disk #2 into the RAID /dev/md0 =
   mdadm /dev/md0 –add /dev/sdb3
   mdadm: added /dev/sdb3
9. Check the RAID status and the rebuild should be started automatically =
   mdadm –detail /dev/md0
   /dev/md0:
   Version : 00.90.03
   Creation Time : Thu Sep 22 21:50:34 2011
   Raid Level : raid1
   Array Size : 486817600 (464.27 GiB 498.50 GB)
   Used Dev Size : 486817600 (464.27 GiB 498.50 GB)
   Raid Devices : 2
   Total Devices : 2
   Preferred Minor : 0
   Persistence : Superblock is persistent

   Intent Bitmap : Internal

   Update Time : Thu Jul 19 01:30:27 2012
   State : active, degraded, recovering
   Active Devices : 1
   Working Devices : 2
   Failed Devices : 0
   Spare Devices : 1

   Rebuild Status : 0% complete

   UUID : 72cc06ac:570e3bf8:427adef1:e13f1b03
   Events : 0.1879848

   Number Major Minor RaidDevice State
   2 8 19 0 spare rebuilding /dev/sdb3
   1 8 3 1 active sync /dev/sda3

10. Check the NAS site for the rebuild % progress
11. After the RAID rebuild complete, restart NAS to clean all previous mount point folder for sdb3

Obtained from this link