HARD DRIVE RECOVERY USING LINUX ( KNOPPIX )

[update 6/13/2013]

First read this article and try to understand it. This guide, although good, has too many typos, and too many updates. If your having a hard time understanding this, I apologize. The key thing you need to google for if this just doesnt make sense 1. How to download and run Knoppix on a Virtual Machine 2. How to find out what disks are in your system for linux 3. How to use dd_rescue…

An important note: since writingarticle I have been using ddrescue instead of dd_rescue in my own personal clones. ddrescue works alot better in my opinion. In this article I talk about dd_rescue. However I will provide a link for the better version ddrescue. The difference in the name is as such, if it has the underscore its older and not as good. By not as good I dont mean one clean is better then another, I just mean one clone device is better at reading through errors, so if both are 100% successful at the clone both clone would be identical.

dd_rescue is older, ddrescue is newer… So if you read this article and understand it really and feel comfortable with it, read the ddrescue article. It highlights the differences between dd_rescue and ddrescue and why ddrescue is better. Also it shows how to use ddrescue. Read into the links in that article, specifically how the algorithm of ddrescue clones vs how dd_rescue clones(linearly from front to back or back to front). Also ddrescue can be restarted from where it left off unlike dd_rescue, but only if you use a logfile. A log file with dd_rescue is just useless, its only used to see how much of the clone has been done, how many errors were encountered etc.

Article: https://sites.google.com/a/infotinks.com/main/linux—dd_rescue-vs-ddrescue

[update 6/12/2013]

In no particular order we: 

1. install knoppix (also make sure you know what serial number is your source and destination disks)

2. start knoppix in vm 

3. load the disks in usb enclousures 

4. to find what disks your system sees

CHECK DISKS: “fdisk -l“

SEE DETECTED DISKS: “cat /proc/partitions“

5. find out what the serial numbers of those disks are and identify which one is your source and destination clone

SINGLE DISK SERIAL NUMBER: “smartctl -a /dev/sda | egrep -i ‘model|serial’” change the sda to match the disk you want to find out the serial number of.

SERIAL NUMBERS OF ALL DISKS IN PLACE: “for i in a b c d e f g h i j k l m n o p q r s t u v w x y z; do echo “—information on disk sd$i—“; smartctl -a /dev/sd$i | egrep -i ‘model|serial’; done;” 

6. with step 4 we find out which disk is our source and destination drive

7. we begin the clone: “sudo dd_rescue /dev/ /dev/[destination]” 

8. if there were alot of errors (over 100) we do a reverse clone “sudo dd_rescue -r /dev/ /dev/[destination]” 

9. When we are done gracefully shutdown the system “shutdown -h now“

NOTE: The step numbers in the actual guide are different

NOTE: most of the commands need to be prefixed with “sudo” because only admins can write them

 

[update 5/5/2013]

This is specifically for any type of NAS so this is useful with the NETGEAR Readynas when doing Hard drive recovery.

 

[update 2/5/2013]

READ THE GUIDE THOROUGHLY FIRST. When hooking up disks to the computer make sure to use USB Enclosures for VMWARE Player or VirtualBox installation of Knoppix. VMWARE PLAYER and VirtualBox will not recognize disks connected via SATA cables to the computer – it will but not in the way you want. If using a desktop setup – just putting in the live DVD or live USB into a live unit – then you can attach the disks via SATA cables and/or Drive USB Enclosures. You can attach the disks via SATA and try to run it on VM – if you want to figure it out, and possibly risk your data – I can point you in the right direction: You need to have an enterprise edition of VMWARE like VMWARE Workstation and use the SATA disks as Physical Disks – again not recommended unless your a PRO with it.Therefore I just recommend using VirtualBox or VMware player with USB Enclosures its the easiest. Also note that USB Enclousures work wonderfully with VirtualBox, you have the options on the VM to enable USB and USB 2.0, just go into the settings of your VM when the VM is off.

[update 2/6/2013]

In the guide I talked about ‘hd’ and ‘sd’ and the difference between them and if you will see ‘hd’ or ‘sd’ in the system. ‘hd’ will be for IDE disks, ‘sd’ for SATA disks. So on SPARC units from the older READYNASes you will see ‘hd’.

DISCLAIMER: There is no guarantee this will recover your data. Do not do this if you think there is a guarantee.

LENGTH OF PROCESS: Some disks take a few hours to clone, some disks can take up to a month to clone only to find out it was all a fail, or it can also be a success. Its not good to start the day of pessimistically, So let’s hope everything will turn out for the best (short times to clone and success at the end). I have seen disks that took a month to clone and it was successful, so do not think long time equals fail.

 

WHATS GOING ON

We are now at the point where we know some drives need to be cloned. And we know what serial numbers are those drives because the NETGEAR tech has told us. So from the NETGEAR tech we need to have 2 maybe 3 things:

1st – Serial numbers of the bad disks we must clone

2nd – This instruction guide

3rd optional thing – Maybe some RMA number of the new disks coming to get put in place of the bad disks

Also we need some clean new disks, freshly bought from the store, or freshly arrived from the NETGEAR RMA process.

Some Stickynotes and a computer with an internet connection wouldn’t hurt either, infact it’s a must.

Note we are cloning the disks because there is some hope of recovery, the bad disks may have reallocated sectors and ATA errors, luckily cloning doesn’t clone electronics errors like those.

 

NOTE ON A FEW TERMS IN THE BELOW PROCEDURE:

dd_rescue: this is the program that will clone our disks, it’s a Linux based program.

linux: its an operating system. Its actually the OS that is running the guts of the NAS.

knoppix: Its a free distribution of the linux OS which is useful in recovering systems and doing troubleshooting.

‘hd’ vs ‘sd’: Some Linux systems see disks as sd(serial disks or scsi disks) or as hd(hard drives or hard disks)

prompt: This is when you are sitting in front of the Linux operating system and the monitor is either giving you a # or $ meaning that you can type a command. If your not presented with a prompted you cant really type commands persay.

kernel: Its the brain of any operating system

partitions: Disks are split into these, and we don’t give much notice to these in here besides that we are very keen that you must ignore them when you see them.

switch: a switch can mean a lot of things but when I say it here, I mean the following. Every Linux command has its general simple form for example “ls” lists every file in the current working directory. Well a switch is an option that modifies how the command runs and usually what it outputs on the screen. So a switch is basically an option. You type them with a a minus sign usually – or a double minus sign –. Luckily we don’t have any — in this guide. You will see that I use the – I switch a lot meaning information, and -r meaning reverse. Note that each switch is unique to its own command so a -r for example could mean reverse on the dd_rescue command, but ran on another command could mean remove or recursive.

 

OVERALL PROCESS – START HERE

(STEP 1) Turn off the NAS

(STEP 2) Pull out all of the disks. Label the disks with stickynotes with the slot number they came from. Put a big “C” on the disks that must be cloned. and Put “Good” on the stickynotes going with disks that are fine.

(STEP 3) Clone all the disks that must be cloned in your desktop/VM, the ones with the “C” stickynote on them. Clone them 1 by 1. This step is easier said then done, so I have layed it out simply below in section “===STEP 3: CLONING===”. The source disks are the ones with the C, the destination disks will be labeled “NEW” on the sticky note when the clone is finished with it.

(STEP 4) Put back in the NAS the “C” and “NEW” disks in the same order as before. So again put back in the NAS the old good disks and the new clones, and remember to keep order – thats why we numbered the slot numbers on the sticky notes.

(STEP 5) Turn on the NAS in normal boot and wait for it to come up and sync up, this process could take a couple of hours, the process will be visible through the logs, (If front view is accessible go to System Tab – Status subtab – Logs and hit the blue DOWNLOAD ALL LOGS button, download and unzip them, and open up mdstat log, it should have a percentage and a speed value on the resync process and a time to finish value). Also the process could be visible in RAIDar

 

Note: on the stickynote you should have the slot number written and either a “C”, “GOOD”,or “NEW”

 

STEP 3: CLONING

 

BEFORE WE START:

Here is the actual cloning guide for each pair of source and destination disks. The source disks should have a big “C” on the sticky note associated with them and also a slot number to tell you what slot they belong in. Attach a sticky note to the disk that will be the destination disk(its the new freshly bought or freshly arrived from RMA disk) and just put the slot number on it. When the clone is done we will put “NEW” on the sticky note.

 

PREPARE YOUR COMPUTER:

Go to Google and search for Knoppix 6.2 Live CD. Download the cd image that goes along with your language of choice and architecture of your PC. This of course is all located at www.knoppix.com.

Do not download the DVD that 4.5 GB of something we don’t need, we only need the 600 or so MB package which houses the Knoppix live cd. Burn that software to a CD and boot up your computer with it, or start a Virtual Machine with something like VMWare Player (also a free software that can be Googled for). Or What I recommend is Virtual Box by Oracle – also free software.

Note on Disk attachment:

If using VM – like Virtual Box or VMWARE Player: Use USB Enclosures for simplicity and no risk of data – SIDE NOTE:  SATA cables are not recognized by the VM/Guest system unless you use enterprise VM software like VMware Workstation and use the SATA disks as “Physical Drives” ( I dont recommend doing the “Physical Drives” setup with the VM enterprise programs as you must be a Pro with VMs to not hurt your Data) 

If using a full desktop setup (by simply popping in the live USB or CD/DVD into your PC and booting your PC into that OS) – Use SATA cables or USB Enclosures – either way the system will recognize them

 

THE CLONING STEPS:

 

(step 1)

Boot up the PC or the VMware with the Knoppix Live CD. Do not plug in any of the source or destination drives yet.

 

(step 2)

Launch a Terminal Window (there should be an Icon or Right Click on the Desktop with your mouse and select Terminal). A command prompt looking window should come up where you could write commands that could either A make everything work or B break everything, so be careful and follow the syntax(spelling) of my next commands listed here to the T.

 

(step 3)

First we need to identify what disks the system sees by default and whether it sees disks as ‘hd’ or ‘sd’. The disks Linux should see right now are just your regular disks that are used with your computer. So to start type “cat /proc/partitions” (Type that without the quotes. I will put all the commands within quotes, do not type the actual quotation marks, only the text within them. Hit enter after you type them). Next you will see all the disks that your system currently sees and also the partitions associated with them, we do not care about the partition entries (they are the ones ending with number). Most likely the system will see all your disks as ‘sd’ instead of ‘hd’.

 

EXAMPLE 1:

Here is an example of some output on a similar Linux system – NOTE YOUR OUTPUT WILL BE MUCH SIMPLER LOOKING THEN THIS HOPEFULLY, and if it looks similar then all this might be a little too easy for you:

# sudo cat /proc/partitions

major minor  #blocks  name

   7        0      81704 loop0

   8        0 1953514584 sda

   8        1    4194304 sda1

   8        2     524288 sda2

   8        3 1948793912 sda3

   9        0    4193268 md0

   9        1     524276 md1

   9        2 1948792704 md2

253        0 1938292736 dm-0

   8       16   15859712 sdb

   8       32     125440 sdc

   8       33     125333 sdc1

 

Notice that my system sees things called ‘loop’s, ‘sd’s, ‘md’s, ‘dm’s. We only care about the ‘sd’s in this case (So now I know my system sees disks as ‘sd’ instead of ‘hd’, so that clears that up). Notice how the system sees sda,sda1,sda2,sda3,sdb,sdc,sdc1. Again we do not care about the numbered entries those are just the partitions, so take out the numbered entries so that we are left with sda,sdb,sdc. That means that currently I have 3 disks in the system weather its via USB or via SATA cable. So now I have cleared up 2 questions. Does my system see disks as sd or hd? sd. What are the disks that my system sees regularly? sda,sdb and sdc. Again your system will be a lot simpler than this. You will probably only see something like this:

 

EXAMPLE2:

# sudo cat /proc/partitions

major minor  #blocks  name

8        0 1953514584 sda

8        1 1953514580 sda1

Meaning that your system also sees disks as ‘sd’s and not ‘hd’ and that you only have 1 disk in the system reguraly which is disk sda.

 

NOTE ON THIS ‘sd’ or ‘hd’ BUSINESS: All of the examples below will be with ‘sd’, but maybe you will have a system that sees disks as hd, well if it does then thats not problem. All you have to do is everywhere you see sd in my example commands replace it with hd.

 

NOTE ON PLUGGING DISKS IN: You can plug in the disks via SATA cables that you could have as extra connections or simply using a USB external chassis of a sort works as well. The different grades of USB version will just forsee the speed of the cloneing.

(step 4) 

Plug in the source disk (labeled with a C on the stickynote) we are working on and run the “cat /proc/partitions” command again, you will see that there is a new entry in there. For example look at the above EXAMPLE2 and when I plug in the source disk it will be:

EXAMPLE2a:

# sudo cat /proc/partitions

major minor  #blocks  name

8        0 1953514584 sda

8        1 1953514580 sda1

8        0 1953514584 sdb

8        1    4194304 sdb1

8        2     524288 sdb2

8        3 1948793912 sdb3

 

Ignore all the numbered entries like sda1,sdb1,sdb2,sdb3 (which just lists partitions, which will be recreated with the cloning process either ways, but that doesn’t matter to us now or ever, so forget I said that, its just a little FYI) So now we know that sdb is the source disk. Simply because before we added in the source drive we only saw sda, being the drive your computer comes with, and then after we plugged in the source drive which added an extra entry sdb. So sdb has to be the source drive since that entry came up right as we plugged in, right? Well You can verify the serial numbers with this this command “smartctl -i /dev/sdb” (THE GENERAL FORM OF THE COMMAND: “smartctl -i /dev/sdX” or “smartctl -i /dev/hdX“, X being the letter associated with the disk and ‘sd’ or ‘hd’ based on if your system sees disks as ‘sd’ or ‘hd’). Another helpful command at our disposal is hdparm run it like so “hdparm -i /dev/sdb” (THE GENERAL FORM OF THE COMMAND: “hdparm -i /dev/sdX” or “hdparm -i /dev/hdX”, X being the letter associated with the disk and ‘sd’ or ‘hd’ based on if your system sees disks as ‘sd’ or ‘hd’). Both of these commands will let you know the serial number associated with them.

 

A LITTLE NOTE ON THE SERIAL NUMBER LISTED ON THE DISKS VS HOW ITS LISTED ON THE LINUX KNOPPIX KERNEL: serial numbers that are printed on the physical drive could be much longer or have spaces or dashes in different

places so for example a drive that shows a serial number of ABCDEF on hdparm or smart-error tools (smartctl) could actually be labeled like AS-DFABC-DEFA-SDFXC, note the ABCDEF in the middle. In reallife this will not be that drastic of change, where you have to look for the embedded serial number with a magnifying glass etc.

 

(step 5)

Repeat step 4 with the destination drive(plug in destination drive and repeat the commands to identify how linux names the disk). Note the name/id it got from the Knoppix kernel using the same commands.

EXAMPLE2b:

Here is EXAMPLE2b to go with our EXAMPLE2 and EXAMPLE2a. Where originally we only had disk sda and then when we added the source disk we saw it as sdb. So plugging in the destination drive I see the following output with the “cat /proc/partitions” command:

# sudo cat /proc/partitions

major minor  #blocks  name

8        0 1953514584 sda

8        1 1953514580 sda1

8        0 1953514584 sdb

8        1    4194304 sdb1

8        2     524288 sdb2

8        3 1948793912 sdb3

8        0 1953514584 sdc

Again ignore the numbered entries being sda1,sdb1,sdb2,sdb3. Notice how the third drive doesnt have any partitions, well that doesnt matter, I just didnt include them, your job is to ignore them either way, so you shouldn’t of noticed that sdc didn’t have a numbered entry to go with it.

 

(step6)

So now we should know the disk name/id that goes with the source and destination drive. To verify our judgment is right there is this helpful “bash miniscript” you can type up. Just make sure you get the syntax perfectly with all the spaces, commas, semicolons, what’s upper and what’s lowercase, etc. However note everything so far involved lower case commands

 

For a system using ‘sd’ based drives (**):

# sudo for i in a b c d e f g h i j k l m n o p q r s t u v w x y z; do echo “—information on disk sd$i—“; smartctl -i /dev/sd$i |     egrep -i ‘model|serial’; done;

For a system using ‘hd’ based drives (**):

# sudo for i in a b c d e f g h i j k l m n o p q r s t u v w x y z; do echo “—information on disk hd$i—“; smartctl -i /dev/hd$i | egrep -i ‘model|serial’; done;

 

EXAMPLE3 (This is the output if I would of ran the above sd miniscript at the time we were at EXAMPLE2, not EXAMPLE2a and not EXAMPLE2b)

Note the following output is from a system with just one disk

# sudo for i in a b c d e f g h i j k l m n o p q r s t u v w x y z; do echo “—information on disk sd$i—“; smartctl -i /dev/sd$i | egrep -i ‘model|serial’; done;

—information on disk sda—

Model Family:     Hitachi Ultrastar A7K2000

Device Model:     Hitachi HUA722020ALA330

Serial Number:    JK11A1YAG962PV

—information on disk sdb—

—information on disk sdc—

—information on disk sdd—

—information on disk sde—

—information on disk sdf—

—information on disk sdg—

—information on disk sdh—

—information on disk sdi—

—information on disk sdj—

—information on disk sdk—

—information on disk sdl—

—information on disk sdm—

—information on disk sdn—

—information on disk sdo—

—information on disk sdp—

—information on disk sdq—

—information on disk sdr—

—information on disk sds—

—information on disk sdt—

—information on disk sdu—

—information on disk sdv—

—information on disk sdw—

—information on disk sdx—

—information on disk sdy—

—information on disk sdz—

From this output we learn that the system only sees 1 disk and that disks serial number is JK11A1YAG962PV and that the disk is sda.

 

So run the above ‘sd’ or ‘hd’ miniscript and make sure what you believe the source disk is and what you believe the destination disk is, line up with the results of that miniscript. If anything trust the miniscripts output over what you had before since this is the most recent.

 

(step7) 

Now we can begin the cloning. All steps 3 through 6 were, were just to ensure that you know what name/id your system sees your source disk and your destination disk. Again at this point your source disk should have a big “C” on its stickynote along with the slot number, and the destination disk shouldnt have anything but a number. That number should match the slot number from the source disk, That number tells you what slot this disk will go after the cloning is hopefully successfull.

 

Once you know which drive is which you can start the clone process with the dd_rescue command. The command is of the form “sudo dd_rescue /dev/<source disk> /dev/<destination drive>“

For ‘sd’ systems: “dd_rescue /dev/sdX /dev/sdX” wher the left sdX is the source disk and the right sdX is the destination drive and the X is the letter that should be different on the right and on the left.

For ‘hd’ systems: “dd_rescue /dev/hdX /dev/hdX” wher the left hdX is the source disk and the right hdX is the destination drive and the X is the letter that should be different on the right and on the left.

 

EXAMPLE2c:

This is to go along with EXAMPLE2, EXAMPLE2a and EXAMPLE2b from above

“sudo dd_rescue /dev/sdb /dev/sdc“

               

Now we wait: weather its a few hours or a few days or almost a month. It will go over errors a few times to try and recover the data.

 

(OPTIONAL step 8)

If you see a lot of errors or see that there is no more data being shown as ‘succxfer:’, then it probably means the drive got marked faulty by the linux kernel. At this point its time to try the reverse cloning operation. So reboot the system with the “reboot” command (or force the reboot with “reboot -f”, if you want to risk forcebly rebooting the system, its not too bad to force it if the step7 command is stopped and your welcomed with the prompt again, which if you are at this step, you are okay to run “reboot -f”.) When the system comes back up, it is very possible the numbering of the disks got reordered by the kernel run the “cat /proc/partitions” command again to ensure you see the source and destination drive and verify you know the name/id of the source and destination drive. You can use the bash miniscript from above to be really sure, again thats the command from above that I have stared with (**). Then when we know what the source and destination disk name is we can beging the reverse cloning. Its done with the dd_rescue command however you must add the -r switch to the command indicating you are cloning from the end of the drive to the front.

 

The reverse command is of the form “sudo dd_rescue -r /dev/<source disk> /dev/<destination drive>“

For ‘sd’ systems: “dd_rescue -r /dev/sdX /dev/sdX” wher the left sdX is the source disk and the right sdX is the destination drive and the X is the letter that should be different on the right and on the left.

For ‘hd’ systems: “dd_rescue -r /dev/hdX /dev/hdX” wher the left hdX is the source disk and the right hdX is the destination drive and the X is the letter that should be different on the right and on the left.

 

EXAMPLE2d:

This is to go along with EXAMPLE2, EXAMPLE2a, EXAMPLE2b, and EXAMPLE2c from above.

Assuming that after the reboot the drives didnt get reordered and that sdb is still the source drive and sdc is still the destination drive.

“sudo dd_rescue -r /dev/sdb /dev/sdc“

 

(step 9)

 After the clone is done label the destination drive’s sticky note with the word “NEW” and continue with the general steps back above in the “OVERALL PROCESS – START HERE” section

NOTE: When your done cloning, you can gracefully shutdown Knoppix like this: “sudo shutdown -h now“. Note only shutdown Knoppix if there are no further clones needed to be done.