Note this is not really a clone, clones work more with images in my opinion. They work directly on the block level instead of the file level.
I give you this which my own personal backup strategy for my external drives. I dont mind if you know this, I changed some of the words in my directories. Big woop.
One thing I dont cover is how I setup my scheduled tasks with Windows Task Scheduler, because I hate describing click by click you can google your way to how to set it up. I just used the default options that made sense and I run this script a couple times a week on Tuesday and Sunday.
4 Sections here:
1. This intro so this is the end of section 1
2. How to setup the main backup scripts, the actual meat of the backup job
3. OPTIONAL watch progress
4. OPTIONAL setup a progress script for the very first job (considering that the target was first empty, either way measures speed good)
RSYNC BACKUP FOR 2 EXTERNAL DRIVES – DRIVE TO DRIVE RSYNC WITH LOGS
###################################################################
REQUIREMENTS:
————-
(1) 2 drives and the destination should be bigger or equal to the source drive (or at least have as much space as there is data – its not very logical to have a smaller destination unless we are only backing up a small subset of the source, but here we are not its a simple slap on the data from drive 1 to drive 2… eh and exclude 1 little folder).
(2) A Windows PC
(3) Cygwin installed with the RSYNC package (Meaning install Cygwin and when you do so it will ask for a list of packages/programs to install with it, so pick rsync)
(4) Also install Notepad++ and write all the scripts in it with Linux mode even the bat file(as the bat file only has 1 line it doesnt matter if it ends in the linux line feed, more on this later)
(5) Dont start troubleshooting these and testing them if you have other long important rsync commands runing on this system because during troubleshooting you might need to run mass destroy all currently running rsync commands
INTRO:
——
Both drives are 4TB drives, I save all of my data to drive E and know that my F drive will have a good backup with this.
I exclude the backingup of the log files as they are open for writing during the process and want to skip that. For ease I just skip the folder where all of the scripts and logs lay for this job.
I schedule the Task with Windows Task Scheduler to run the
e2f-start-from-windows.bat file which starts the e2f-start.sh bash script (which cygwin runs) that file starts the e2f-backup.sh(this is the actual file that has the meat of the rsync scripts) and saves all of the output generated by the backup to some logs with dates.
In cygwin the drives locations are like this, the E: drive is at /cygdrive/e/ which is my source 4TB drive and the destination is F: which is /cygdrive/f/. All of the scripts lay in the source drive in the following location E:\1DailyBackupOfDocuments\Simba2UpBack\ and thats the folder I dont want to be backed up (again so I dont get errors while it encounters afile thats open for writing, because we are obviously writing logs to it at the same time that we would be backing it up – we dont have a copy on write filesystem here). Note althought I exclude E:\1DailyBackupOfDocuments\Simba2UpBack\ and everything inside it, we still get to include E:\1DailyBackupOfDocuments\ and everything else inside it.
Here is the lay out of the logs and script folder, I will then show what I each file has in it and that will sum it up.
So here is the folder we are not backing up and all of the scripts in it – this will be listed in Windows Form first and then in linux/cygwin form:
E:\1DailyBackupOfDocuments\Simba2UpBack\e2f-excludes1
E:\1DailyBackupOfDocuments\Simba2UpBack\e2f-backup.sh
E:\1DailyBackupOfDocuments\Simba2UpBack\e2f-start.sh
E:\1DailyBackupOfDocuments\Simba2UpBack\e2f-start-from-windows.bat
E:\1DailyBackupOfDocuments\Simba2UpBack\[logs will be here after the jobs are ran]
Note that case sensitivity is ignored by windows but is payed attention to by linux/cygwin, thus this is case sensitive – the weaker one wins, or the more restrictive option usually always wins in the computer world.
/cygdrive/e/1DailyBackupOfDocuments/Simba2UpBack/e2f-excludes1
/cygdrive/e/1DailyBackupOfDocuments/Simba2UpBack/e2f-backup.sh
/cygdrive/e/1DailyBackupOfDocuments/Simba2UpBack/e2f-start.sh
/cygdrive/e/1DailyBackupOfDocuments/Simba2UpBack/e2f-start-from-windows.bat
/cygdrive/e/1DailyBackupOfDocuments/Simba2UpBack/[logs will be here after the jobs are ran]
I will go over whats inside from smallest to biggest file.
Some things to note about the scripts:
————————————–
* Alot of extra notes here if anything just read the VERY IMPORTANT one at the bottom
* The exclude script doesnt just exclude the folder with the logs, it also excludes some of the system default files that are associated with every drive. Like the “System Volume Information” folder and the “$RECYCLE.BIN” folder which is actually not normally seen as that but more like just the REcycle Bin we are all aware of. Also I have The virus scanner AVG which makes a “$AVG” folder at the root of every drive because AVG devs know Moving is faster then copying. So I also exclude the $RECYCLE.BIN and the $AVG and the System Volume Information from the copy. Its interesting to note that the Volume Infomation folder even though it has spaces in it, didnt need to have them escaped with \ as linux usually requires, and I didnt need to put SYstem Volume Information in quotes, it means the exclude engine with RSYNC is smart.
* Not for all of the files I minimize on the whilespace before and after the scripts, I dont want any extra spaces or empty uneeded lines.
* How do you know if the backup is still running since its all from the back ground? Well you run “ps” and if you see at least 1 rsync entries it means its running. Usually rsync runs 3 processes per one execution to speed up IO.
* Note setting up the scripts you might need to troubleshoot the e2f-start.sh and since the rsync script e2f-backup.sh command is nohuped into the background, here is how you kill it first see the process is alive with “ps” and kill the process with “taskkill /F /IM rsync*”, More info here: https://sites.google.com/a/infotinks.com/main/cygwin—killall-type-of-command Note this will kill all rsync commands so if you have running rsync commands opps.
* Infact make an option troubleshooting file called “STOP ALL RSYNCS.bat” make sure it has WINDOWS LINE ENDINGS that end with CR+LF, that can be done by making this file with Regular Notepad or doing the convert feature of Notepad++->Menu:Edit->EOL Conversion->Windows. Script is below.
* Note running e2f-start-from-windows.bat starts the backup and opens the command prompt window. IF that black command prompt window is closed at any time the rsync job will stop running. How did I find this out is pretty cool, while running CYGWIN in one window and
* Note some documentation says to point the Windows PATH variable to the CYGWIN bin folder well I didnt do that and everything is fine, why? because we call upon everything with full path structures, besides a few commands like “taskill” but thats a windows common command and “date” well that gets called from the linux cygwin bash script so it doesnt operate under the windows PATH rules. So inside cygwin there is a seperate PATH variable that is not the same as the windows PATH variable. Just incase you are wondering the PATH variable is the variable that tells windows or cygwin into which folder to look when commands are typed so for example when I type rsync in windows it doesnt work, why? because the PATH variable doenst point to where rsync sits, but in cygwin it works why? because cygwins path variable does. Some articles want you to include CYGWINS bin directory in windows PATH so you can call cygwin commands from the black command prompt window, so I would be able to run “rsync” for example from “cmd”
* In the backup.sh script I call upon the powers of windows iacls which fixes permissions by simply reseting them essentially to all access, (windows 8 has a terrible way of breaking permissions well this fixes it). I use the varaible `WINDEST1=”F:\\”` All I wanted to say is the two slashes are not a type. A slash \ is a special character in linux and I needed to escape it to write one. (Which boggles my mind because I didnt do it in my other backup article and everything still worked) Anyhow later in the backup.sh script its called like this: `icacls “$WINDEST1” /T /C /Q /RESET` which resets the permissions
* VERY IMPORTANT – Make sure every file ends in a Linux Line feed LF instead of a Windows/DOS line feed CR+LF: (http://en.wikipedia.org/wiki/Newline) There is linux and windows mode in regards to how a line ends and new line begins. At the end of every line there is CR character and an LF character in Windows know as CR+LF, note they are not really letters. In linux its just LF. (I believe in the old MAC its just CR but in OSX its LF just like Linux)The files must have new lines suitable for linux or else none of them will work. The windows bat file is only 1 line long thus if its left in linux mode thats okay. So to simplify what I am saying: each one of the scripts needs to end in the linux form LF. The CR+LF is not needed and will cause errors. So if you wrote the scripts in Notepad then they are all in CR+LF and thats not going to work, we need to convert them all to LF line endings. The easiest way to do it is with Notepad++, link is above for download, go to the “View” menu and select “Show Symbol” and then select “Show All Character” or just end of line. If all of files end in LF then your good to go, if you have a file that every line ends in CR+LF then. Note its extremely rare for a file to have some lines end in CR+LF and some in LF, well for 99.99% of the time its all the same – I never seen a file have multiples types of line endings. So if you have lines ending in CR+LF here is how to convert to LF – besides using many software out there like dos2unix (also unix2dos for the vice versa method does exist incase you are curious), Notepad++ can convert us: Go to the “Edit” menu and hit “EOL Conversion” and hit “UNIX/OSX Format”
QUICK RECAP
———–
* Install Cygwin and Notepad++
* Copying from E: to F: drive, in cygwin terms from /cygdrive/e/ to /cygdrive/f/
* Here are the script files in my case, and I will exclude these from being backed up because of the e2f-excludes1 file
E:\1DailyBackupOfDocuments\Simba2UpBack\e2f-excludes1 = this file tells what not to backup
E:\1DailyBackupOfDocuments\Simba2UpBack\e2f-backup.sh = this is the main backup script, with the rsync commands, also an icacls command to reset(And fix permissions) at the end as windows 8 has messed up permission issues
E:\1DailyBackupOfDocuments\Simba2UpBack\e2f-start.sh = this starts the backup script
E:\1DailyBackupOfDocuments\Simba2UpBack\e2f-start-from-windows.bat = this is the windows executable, windows cant start the .sh files, but this will use bash to start the .sh files
E:\1DailyBackupOfDocuments\Simba2UpBack\[logs will be here after the jobs are ran]
* One of the excludes is of course the folder with the scripts and logs meaning this one “E:\1DailyBackupOfDocuments\Simba2UpBack\” but dont worry ” E:\1DailyBackupOfDocuments\” is still included with everything ELSE inside it.
* Linux is case senstive so cygwin is and therefor our scripts are as well
* Make sure each script ends in LF line feed instead of CR+LF, to change to LF: when writing scripts in Notepad++ go to Edit menu then EOL Conversion and select UNIX/OSX
*When done troubleshoot each script, first open up CYGWIN and try the e2f-backup.sh script, and then the e2f-start.sh and then from windows open “cmd” and test the e2f-start-from-windows.bat
* When you are troubleshooting you will have to turn off rsync when it launches into the background – the nohup command in e2f-start.sh puts it in the background – so to kill the scripts run this from either the “cmd” or from the CYGWIN windows “taskkill /F /IM rsync*”
* Infact make an option troubleshooting file called “STOP ALL RSYNCS.bat” make sure it has WINDOWS LINE ENDINGS that end with CR+LF, that can be done by making this file with Regular Notepad or doing the convert feature of Notepad++->Menu:Edit->EOL Conversion->Windows. Script is below.
* Finally you might need to go to the destination folder and delete everthing from it (For me its everything because they are going to be exact replicas of files – minus the things that I am excluding with the e2f-excludes file): from CYGWIN type “cd /cygdrive/f/” and then “rm -rf *” but becareful as that will destroy everything in F: drive and also note that if misspelled can delete awhole lot more.
* Make sure not to have alot of whitespace: for that matter no extra new lines at the end of the scripts and no new lines at the top. new lines meaning empty lines.
THE SCRIPTS
===========
OPTIONAL TROUBLESHOOTING SCRIPT – KILL ALL RSYNC.bat – – EOL must be CR+LF
—————————————————————————
D:\cygwin\bin\echo.exe THIS RUNS FOR 10 SECONDS AND KILLS ALL RSYNCS AND DISAPPEARS
D:\cygwin\bin\ps.exe
taskkill /F /IM rsync*
D:\cygwin\bin\ps.exe
D:\cygwin\bin\sleep.exe 10
e2f-excludes1
————-
1DailyBackupOfDocuments/Simba2UpBack/
$AVG/
$RECYCLE.BIN/
System Volume Information/
e2f-start-from-windows.bat
————————–
D:\cygwin\bin\bash.exe -l -c “/cygdrive/e/1DailyBackupOfDocuments/Simba2UpBack/e2f-start.sh”
e2f-start.sh
————
#!/bin/bash
# All of the backup scripts & logs: E:\1DailyBackupOfDocuments\Simba2UpBack
DT=`date “+%D-%T”`
DATE=`echo “$DT” | tr [/:] _`
echo $DATE
LOG=/cygdrive/e/1DailyBackupOfDocuments/Simba2UpBack/e2f-Full-Log-$DATE.Log
ERR=/cygdrive/e/1DailyBackupOfDocuments/Simba2UpBack/e2f-Full-Error-$DATE.log
echo “### $DT ###”
echo “Starting the backing up of drive E (Simba) to F (UpBack) drive”
echo “This Log file will be made, appended with this date:”
echo $LOG
echo “This Error log will be made:”
echo $ERR
nohup /cygdrive/e/1DailyBackupOfDocuments/Simba2UpBack/e2f-backup.sh > $LOG 2> $ERR &
echo
echo “==WARNING:==”
echo “IF RUNNING IN WINDOWS THRU e2f-start-from-windows.bat THEN DO NOT CLOSE THE cmd WINDOW”
echo “OR ELSE JOB WONT FINISH. IF YOU DID ON ACCIDENT CLOSE IT, THEN JUST RESTART IT”
echo “IT WILL PICK UP WHERE IT LEFT OFF”
echo
echo “==NOTE:==”
echo YOU CAN SEE THE PROGRESS by \”tail -f $LOG\”
e2f-backup.sh
=============
#!/bin/bash
# All of the backup scripts & logs: E:\1DailyBackupOfDocuments\Simba2UpBack
RSYNC=/bin/rsync
S1=”/cygdrive/e/”
EXCLUDES1=”/cygdrive/e/1DailyBackupOfDocuments/Simba2UpBack/e2f-excludes1″
D1=”/cygdrive/f/”
WINDEST1=”F:\\”
LOG1=”/cygdrive/e/1DailyBackupOfDocuments/Simba2UpBack/e2f-DocsBackup.log”
SWITCHES=”-ahv –human-readable –stats –progress –delete”
echo “#######################################################”
echo “#######################################################”
echo “#######################################################”
echo
DT1s=`date “+%D-%T”`
echo “$DT1s JOB START TIME”
echo “Starting the RSYNC backup job of Drive E”
echo “Excluding the following from drive E”
cat $EXCLUDES1
echo
echo “#######################################################”
echo
echo “THE COMMAND TO BACKUP E:”
echo $RSYNC $SWITCHES –log-file=”$LOG1″ –exclude-from=”$EXCLUDES1″ “$S1” “$D1”
echo
echo “THE COMMAND STARTS … NOW:”
echo
$RSYNC $SWITCHES –log-file=”$LOG1″ –exclude-from=”$EXCLUDES1″ “$S1” “$D1”
echo
echo “#######################################################”
echo
DT1e=`date “+%D-%T”`
echo “$DT1e JOB FINISH TIME”
echo
echo “#######################################################”
echo “#######################################################”
echo
echo “Recap of Times:”
echo
echo “RSYNC Job Start : $DT1s”
echo “RSYNC Job End : $DT1e”
echo
echo “#######################################################”
echo “#######################################################”
echo “#######################################################” >> $LOG1
echo
Pstart=`date “+%D-%T”`
echo “WINDOWS 8 HAS PERMISSIONS ISSUES: Lets fix em with this command:”
echo “icacls is a windows command – not cygwin – but it works none the less”
echo “/T = recursive, /C = continue on errors and show em”
echo “/Q = dont list every file, /RESET = resets the permissions and fixes em”
echo
echo “STARTING FIX AT: $Pstart”
echo
echo icacls “$WINDEST1” /T /C /Q /RESET
icacls “$WINDEST1” /T /C /Q /RESET
Pend=`date “+%D-%T”`
echo
echo “FIX FINISHED AT: $Pend”
echo
echo “#######################################################”
echo “#######################################################”
echo
echo
echo “Recap of Times:”
echo
echo “RSYNC Job Start : $DT1s”
echo “RSYNC Job End : $DT1e”
echo “PERM. Fix Start : $Pstart”
echo “PERM. Fix End : $Pend”
echo
echo “#######################################################”
echo “#######################################################”
MY FIRST CLONE JOB
##################
There are 3 scripts:
E:\1DailyBackupOfDocuments\Simba2UpBack\e2f-backup.sh = this is the main backup script, with the rsync commands, also an icacls command to reset(And fix permissions) at the end as windows 8 has messed up permission issues
E:\1DailyBackupOfDocuments\Simba2UpBack\e2f-start.sh = this starts the backup script into a nohup of its own
E:\1DailyBackupOfDocuments\Simba2UpBack\e2f-start-from-windows.bat = this is the windows executable, windows cant start the .sh files, but this will use bash to start the .sh files
And a helper file:
E:\1DailyBackupOfDocuments\Simba2UpBack\e2f-excludes1 = this file tells what not to backup
For the first clone I just ran e2f-backup.sh manually from cygwin like this:
cd /cygdrive/e/1DailyBackupOfDocuments/Simba2UpBack
nohup ./e2f-backup.sh &
And followed everything like this:
while true; do echo “=====”; date; (df -h && df) | egrep -i “drive/e|drive/f|file”; tail -n6 nohup.out; sleep 1;done;
AFTER EVERY VARIABLE IS EVALUATED AND ALL OF THE COMMENT ECHO LINES ARE IGNORED THIS IS ALL THE e2f-backup.sh DOES:
rsync -ahv –human-readable –stats –progress –delete –log-file=”/cygdrive/e/1DailyBackupOfDocuments/Simba2UpBack/e2f-DocsBackup.log” /cygdrive/e/ /cygdrive/f/
icacls F:\\ /T /C /Q /RESET
The e2f-exclude1 file has this inside it:
1DailyBackupOfDocuments/Simba2UpBack/
$AVG/
$RECYCLE.BIN/
System Volume Information/
AFTER EVERY VARIABLE IS EVALUATED AND ALL OF THE COMMENT ECHO LINES ARE IGNORED THIS IS ALL THE e2f-start.sh DOES:
nohup ./e2f-backup.sh > some_log_with_date &> some_error_log_with_date &
THE START FROM WINDOWS LOG – e2f-start-from-windows.bat – ONLY DOES THIS:
D:\cygwin\bin\bash.exe -l -c “/cygdrive/e/1DailyBackupOfDocuments/Simba2UpBack/e2f-start.sh”
###################################################################################################
THE FIRST CLONE, SNAPPED AT DIFFERENT TIMES:
###################################################################################################
Sat, Jul 27, 2013 4:05:31 AM
Filesystem Size Used Avail Use% Mounted on
E: 3.7T 2.9T 809G 79% /cygdrive/e
F: 3.7T 56G 3.6T 2% /cygdrive/f
Filesystem 1K-blocks Used Available Use% Mounted on
E: 3906983932 3059393516 847590416 79% /cygdrive/e
F: 3907010372 58405712 3848604660 2% /cygdrive/f
…not showing you the last 3 lines as it shows the files its copying…
==============================================================================
Sat, Jul 27, 2013 7:21:12 PM
Filesystem Size Used Avail Use% Mounted on
E: 3.7T 2.9T 809G 79% /cygdrive/e
F: 3.7T 665G 3.0T 18% /cygdrive/f
Filesystem 1K-blocks Used Available Use% Mounted on
E: 3906983932 3059453580 847530352 79% /cygdrive/e
F: 3907010372 696260892 3210749480 18% /cygdrive/f
…not showing you the last 3 lines as it shows the files its copying…
==============================================================================
Just copy paste the result from one second, wait a few hours or minutes and copy paste it into the same notepad and run some math on it.
Using www.wolframalpha.com You can find out these answers:
SPEED:
——
THE FORMULA:
(((dest used now)-(dest used before))/((time now)-(time before)))
PUT THIS INTO WOLFRAM ALPHA:
((696260892 kilobytes)-(58405712 kilobytes))/((Sat, Jul 27, 2013 7:21:12 PM)-(Sat, Jul 27, 2013 4:05:31 AM))
Answer: 11.61MB/s
HOURS BETWEEN BOTH POINTS:
————————–
FORMULA:
((time now)-(time before)))
WHAT I PUT IN TO WOLFRAMALPHA:
(Sat, Jul 27, 2013 7:21:12 PM)-(Sat, Jul 27, 2013 4:05:31 AM)
ANSWER:
15.26 hours
DEST CHANGED IN USED:
———————
* Just curious how much the destination changed in size – must be the growing logs and some other stuff.
FORMULA:
(dest used now – dest used then)
WHAT I PUT IN TO WOLFRAMALPHA:
(3059453580-3059393516) kilobytes
ANSWER:
60 MB
TIME LEFT:
———
FORMULA:
(space left to copy)/(current speed), note we already calculated the part after the division /, this part: (((dest used now)-(dest used before))/((time now)-(time before))) was 11.61 MB/s
FORMULA EXPANDED:
(source used-dest used now)/(((dest used now)-(dest used before))/((time now)-(time before)))
WHAT I PUT IN TO WOLFRAMALPHA:
((3059453580 kilobytes)-(696260892 kilobytes))/(((696260892 kilobytes)-(58405712 kilobytes))/((Sat, Jul 27, 2013 7:21:12 PM)-(Sat, Jul 27, 2013 4:05:31 AM)))
ANSWER:
56.54 hours. 2.356 days
SPACE LEFT TO COPY:
——————–
(source used-dest used now)
((3059453580 kilobytes)-(696260892 kilobytes))
2.363 TB
PROGRESS EVEN BETTER THEN ABOVE DOES THE MATH FOR YOU INSTEAD OF WOLFRAMALPHA
#############################################################################
EXAMPLE OUTPUT:
===============
* I trimmed the white space so it fits better here
Sat, Jul 27, 2013 11:18:12 PM
===DF===
Filesystem Size Used Avail Use% Mounted on
E: 3.7T 2.9T 809G 79% /cygdrive/e
F: 3.7T 726G 3.0T 20% /cygdrive/f
Filesystem 1K-blocks Used Available Use% Mounted on
E: 3906983932 3059552052 847431880 79% /cygdrive/e
F: 3907010372 761199672 3145810700 20% /cygdrive/f
===LAST 6 LINES OF THE JOB===
…some personal docs, you dont need to see what files I am copying…
===JOB INFO:===
TARGET: 743360 MB, 725.938 GB
SOURCE: 2987844 MB, 2917.82 GB
REMAINING 2244484 MB, 2191.88 GB – PERCENT DONE: 24.8795 % <- complete if close to 100%===SPEED AND TIME ANALYSIS ESTIMATIONS BETWEEN LAST 2 RUNS===
IN THE LAST [2 SECS] [28672 KBYTES] WERE TRANSFERED @ A RATE OF [14336 KB/s = 14 MB/s = 114688 Kb/s = 112 Mb/s]
–TIME REPORTS IN SECONDS–
ETA IS [160320 s – calc from (KB)/(KBPS)] = [160320 s – calc from (MB)/(MBPS)]
RUNNING TIME IS [53097.1 s – calc from (KB)/(KBPS)] = [53097.1 s – calc from (MB)/(MBPS)]
–TIME REPORTS IN HOURS–
ETA IS [44.5333 h – calc from (KB)/(KBPS)] = [44.5333 h – calc from (MB)/(MBPS)]
RUNNING TIME IS [14.7492 h – calc from (KB)/(KBPS)] = [14.7492 h – calc from (MB)/(MBPS)]
===CPU LOAD AVERAGES:===
0.00 0.00 0.00 1/8
===MEMORY USAGE===
MemTotal: 16717684 kB
MemFree: 13040436 kB
SwapTotal: 3145728 kB
SwapFree: 3040080 kB
===WHATS STILL RUNNING===
IS c[p] -Rnv STILL RUNNING: NO
IS rsyn STILL RUNNING: YESIS ta[r] STILL RUNNING: NO
HOW I DID THAT:
===============
The script full path: /cygdrive/e/1DailyBackupOfDocuments/Simba2UpBack/e2f-OPTIONAL-progress.sh
Whats this? Show the best type of progress, it has to be run with a watch or a while loop to get best results make the time difference about 2 to 3 seconds minimum, 10 seconds for better estimates.
How I run this with the script above, Note this works best on the first clone when the TARGET starts off completely empty. Even if its not empty to start thats okay, this is still a good measure of backup speed.
You can manually run it like this:
cd /cygdrive/e/1DailyBackupOfDocuments/Simba2UpBack/
./e2f-OPTIONAL-progress.sh
then wait 2 seconds
./e2f-OPTIONAL-progress.sh
then wait 2 seconds
./e2f-OPTIONAL-progress.sh
Or :
while true; do echo; echo =============; ./e2f-OPTIONAL-progress.sh; sleep 2; done
Note the extra echos for lines of seperation, I would make the === be longer, like this:
while true; do echo; echo =============================================; ./e2f-OPTIONAL-progress.sh; sleep 2; done
TO GET THIS: Make script below
Make sure this file also ends with LF and not CR+LF
TO DO THAT: With notepad++ go to Edit->EOL Conversion->UNIX
e2f-OPTIONAL-progress.sh
========================
#!/bin/bash
# this script measures progress just modify a couple of variables to match your backup job
# Also it generates to small temp files /tmp/time1 and /tmp/target1
# this works best when the target starts off completely empty – its okay if you started this late thru the job
######### JUST MODIFY THIS SECTION ############
SOURCE_PHRASE_FROM_DF=”drive/e” # MUST CHANGE THIS FOR YOUR JOB
TARGET_PHRASE_FROM_DF=”drive/f” # MUST CHANGE THIS FOR YOUR JOB
SCRIPT1=”c[p] -Rnv” # modify with the output of “ps” or “ps aux”, usually leave it alone
SCRIPT2=”rsyn” # modify with the output of “ps” or “ps aux”, usually leave it alone
SCRIPT3=”ta[r]” # modify with the output of “ps” or “ps aux”, usually leave it alone
FILE_TO_TAIL=”nohup.out” # MUST CHANGE THIS FOR YOUR JOB, USE FULL PATH TO FILE, I GOT AWAY WITH RELATIVE PATH BECAUSE THE NOHUP.OUT IS IN THE SAME DIRECTORY AS I RAN THIS FILE FROM
###### THE PROGRAM SECTION USUALLY DOESNT NEED MODIFICATION #########
TIME1=`cat /tmp/time1`
TARGET1=`cat /tmp/target1`
date
echo
echo “===DF===”
(df -h && df) | egrep -i “$SOURCE_PHRASE_FROM_DF|$TARGET_PHRASE_FROM_DF|file”
echo
echo “===LAST 6 LINES OF THE JOB===”
tail -n6 $FILE_TO_TAIL;
echo
echo
echo “===JOB INFO:===”
TARGET_KB=`df -P | egrep -i “$TARGET_PHRASE_FROM_DF” | awk ‘{print $3}’`
TARGET=`df -Pm | egrep -i “$TARGET_PHRASE_FROM_DF” | awk ‘{print $3}’`
TARGET_GB=`echo $TARGET | awk ‘{x=$0; y=x/1024; print y;}’`
echo “TARGET: $TARGET MB, $TARGET_GB GB”
SOURCE_KB=`df -P | egrep -i “$SOURCE_PHRASE_FROM_DF” | awk ‘{print $3}’`
SOURCE=`df -Pm | egrep -i “$SOURCE_PHRASE_FROM_DF” | awk ‘{print $3}’`
SOURCE_GB=`echo $SOURCE | awk ‘{x=$0; y=x/1024; print y;}’`
echo “SOURCE: $SOURCE MB, $SOURCE_GB GB”
DIFF_KB=`echo $SOURCE_KB $TARGET_KB | awk ‘{x=$1; y=$2; print x-y;}’`
DIFF=`echo $SOURCE $TARGET | awk ‘{x=$1; y=$2; print x-y;}’`
DIFF_GB=`echo $DIFF | awk ‘{x=$0; y=x/1024; print y;}’`
PERC=`echo $SOURCE $TARGET | awk ‘{x=$1; y=$2; print y/x*100;}’`
echo “REMAINING $DIFF MB, $DIFF_GB GB – PERCENT DONE: $PERC % <- complete if close to 100%”
echo
echo “===SPEED AND TIME ANALYSIS ESTIMATIONS BETWEEN LAST 2 RUNS===”
TARGET2=$TARGET_KB
SIZE_DIFF=`echo $TARGET2 $TARGET1 | awk ‘{x=$1; y=$2; print x-y;}’`
TARGET1=$TARGET2
TIME2=`date +%s` # CURRENT TIME IN SECONDS SINCE 1970
TIME_DIFF=`echo $TIME2 $TIME1 | awk ‘{x=$1; y=$2; print x-y;}’`
TIME1=$TIME2 # CURRENT TIME BECOMES OLD TIME
SPEED_KBYTES=`echo $SIZE_DIFF $TIME_DIFF | awk ‘{x=$1; y=$2; print x/y;}’` 2> /dev/null
SPEED_KBITS=`echo $SPEED_KBYTES | awk ‘{x=$1; print x*8;}’`
SPEED_MBYTES=`echo $SPEED_KBYTES | awk ‘{x=$1; print x/1024;}’`
SPEED_MBITS=`echo $SPEED_MBYTES | awk ‘{x=$1; print x*8;}’`
ETA_SECONDS=`echo $DIFF_KB $SPEED_KBYTES | awk ‘{x=$1; y=$2; print x/y;}’` 2> /dev/null
TOTAL_SECONDS=`echo $TARGET_KB $SPEED_KBYTES | awk ‘{x=$1; y=$2; print x/y;}’` 2> /dev/null
ETA_SECONDS_M=`echo $DIFF $SPEED_MBYTES | awk ‘{x=$1; y=$2; print x/y;}’` 2> /dev/null
TOTAL_SECONDS_M=`echo $TARGET $SPEED_MBYTES | awk ‘{x=$1; y=$2; print x/y;}’` 2> /dev/null
ETA_SECONDS_H=`echo $ETA_SECONDS | awk ‘{x=$1; print x/3600;}’` 2> /dev/null
TOTAL_SECONDS_H=`echo $TOTAL_SECONDS | awk ‘{x=$1; print x/3600;}’` 2> /dev/null
ETA_SECONDS_M_H=`echo $ETA_SECONDS_M | awk ‘{x=$1; print x/3600;}’` 2> /dev/null
TOTAL_SECONDS_M_H=`echo $TOTAL_SECONDS_M | awk ‘{x=$1; print x/3600;}’` 2> /dev/null
echo “IN THE LAST [$TIME_DIFF SECS] [$SIZE_DIFF KBYTES] WERE TRANSFERED @ A RATE OF [$SPEED_KBYTES KB/s = $SPEED_MBYTES MB/s = $SPEED_KBITS Kb/s = $SPEED_MBITS Mb/s]”
echo
echo “–TIME REPORTS IN SECONDS–“
echo “ETA IS [$ETA_SECONDS s – calc from (KB)/(KBPS)] = [$ETA_SECONDS_M s – calc from (MB)/(MBPS)]”
echo “RUNNING TIME IS [$TOTAL_SECONDS s – calc from (KB)/(KBPS)] = [$TOTAL_SECONDS_M s – calc from (MB)/(MBPS)]”
echo
echo “–TIME REPORTS IN HOURS–“
echo “ETA IS [$ETA_SECONDS_H h – calc from (KB)/(KBPS)] = [$ETA_SECONDS_M_H h – calc from (MB)/(MBPS)]”
echo “RUNNING TIME IS [$TOTAL_SECONDS_H h – calc from (KB)/(KBPS)] = [$TOTAL_SECONDS_M_H h – calc from (MB)/(MBPS)]”
echo
echo “===CPU LOAD AVERAGES:===”
cat /proc/loadavg;
echo
echo “===MEMORY USAGE===”
cat /proc/meminfo | egrep -i “mem|swap”
echo
echo “===WHATS STILL RUNNING===”
echo -e “IS $SCRIPT1 STILL RUNNING:” `if ps aux | egrep -qi “$SCRIPT1” ; then echo “YES”; else echo “NO”; fi`
echo -e “IS $SCRIPT2 STILL RUNNING:” `if ps aux | egrep -qi “$SCRIPT2” ; then echo “YES”; else echo “NO”; fi`
echo -e “IS $SCRIPT3 STILL RUNNING:” `if ps aux | egrep -qi “$SCRIPT3” ; then echo “YES”; else echo “NO”; fi`
echo $TIME1 > “/tmp/time1”
echo $TARGET1 > “/tmp/target1”