If your using cp or tar or scp or cat|ssh to transfer files chances are you dont have a progress bar or even have a clue as to how fast your volume is changing. Of course you can manually run this:
For du date +%s; du -B1; or for df date +%s; df -B1; (to get date in epoch seconds and volume size in bytes) and do the math yourself with a couple of time spaced readings (say 10 seconds or a few minutes or hours apart) to get an idea of how fast your going. The -B1 argument asks for output to be of blocksize 1 (1 byte) meaning everything is reported in Bytes (like wise -B1024 and -Bk report the same numbers). From the man page of du & df:
-B, –block-size=SIZE use SIZE-byte blocks
SIZE may be (or may be an integer optionally followed by) one of fol- lowing: kB 1000, K 1024, MB 1000*1000, M 1024*1024, and so on for G, T, P, E, Z, Y.
The formula is as simple as it gets:
SpeedInBytesPerSecond = (CurrentByteSize-OldByteSize) / (CurrentTimeInEpochSeconds-OldTimeInEpochSeconds)
ds = (s1-s0)/(t1-t0)
Here are 4 scripts that are copy pasteable. First you need to edit the variables mentioned in the comments. Then you just copy paste the commands. The manual ones you repeat while the automatic ones you dont have to repeat as the looping algorithm takes care of that for you.
df auto, df manual, du auto, df manual
You can use the du scripts if you want to monitor the rate of change of its size. If you want to monitor a whole volume you can use df. Note that if a folder has many subfolders and files, du will take a while to calculate so measuring rate of change with du is not recommended in that scenario, instead measure with df – however it will only be useful if thats the only thing transfering into the server. both of these scripts can be used to monitor how quick space is being freed up (space being freed up is a negative speed)
How to use these scripts:
1) pick the script you wish to use based on if you want auto loops or manual looping (when you repeat the command you get the output). Remember folders with many files will take a moment with du. Where as df will give you an overall rate of change of the entire volume rather than a folder.
2) Copy the script from here and paste into your notepad for editing. Preferably something that supports linux line endings such as Notepad++(although that doesnt really matter as the scripts are one line big anyways – if you dont include the header comment)
2) fill out the variable MNTofDEVICEtoMONITOR, INTERVAL or FOLDER,INTERVAL (the none-looping scripts dont have an INTERVAL variable)
3) Select the edited script
3) Copy the edited script to your clipboard by selecting it and pressing CONTROL-C
4) Paste the script and hit enter to run the script. IF your using putty you simply need to paste by right clicking with the mouse inside of the putty terminal. Note that clicking enter is optional and not needed. If you also selected a newline character (they are invisible but signify and emulate the press of an ENTER key) then you wont need to press ENTER (but if you do it wont damage anything, you will just have an extra blank line on the output). If you didnt slect a newline character when you copied the text you will need to press ENTER after you paste the output into your shell/terminal.
DF
######################################
# df #
######################################
# -- DF -- SPEED CALCULATION (auto/loop) -- change variable MNTorDEVICEtoMONITOR to mount point or device name to monitor & change INTERVAL (seconds) - this monitors change of disk used size on the
(MNTorDEVICEtoMONITOR="/mnt"; INTERVAL=10; SYNC=""; while true; do USEDkvk=`df ${SYNC} -P ${MNTorDEVICEtoMONITOR} | tail -n1 | awk '{print $3}'`; TIMEkvk=`date +%s`; echo "##=== DEVMOUNT: $1 - `date` - USED: $USEDkvk kb===##"; (df -P && df -Ph) | grep -e "$MNTorDEVICEtoMONITOR\|Used"; echo $USEDkvk $TIMEkvk $OLDUSEDkvk $OLDTIMEkvk | awk '{DT=$2-$4; SKB=($1-$3)/DT; SMB=SKB/1024; print "NEW_USED: " $1 " KiB - OLD_USED: " $3 " KiB\nNEW_TIME: " $2 " s - OLD_TIME: " $4 " - DELTA_TIME: " DT "s\n---- SPEED: " SKB " KiB/sec, " SMB " MiB/sec ---" }'; OLDUSEDkvk=$USEDkvk; OLDTIMEkvk=$TIMEkvk; sleep $INTERVAL; done;)
######################
# -- DF -- SPEED CALCULATION (manual) -- change variable MNTorDEVICEtoMONITOR to mount point or device name - SYNC="--sync" to sync before speed reading, SYNC="" to not sync before - manually repeat this command and it will give you average speed since last reading
(MNTorDEVICEtoMONITOR="/mnt"; SYNC=""; USEDkvk=`df ${SYNC} -P ${MNTorDEVICEtoMONITOR} | tail -n1 | awk '{print $3}'`; TIMEkvk=`date +%s`; OLDUSEDkvk=`cat /tmp/ddf-oldused`;OLDTIMEkvk=`cat /tmp/ddf-oldtime`;echo "======== DEVMOUNT: $1 - `date` - USED: $USEDkvk kb ========"; (df -P && df -Ph) | grep -e "$MNTorDEVICEtoMONITOR\|Used"; echo "$USEDkvk $TIMEkvk $OLDUSEDkvk $OLDTIMEkvk" | awk '{DT=$2-$4; SKB=($1-$3)/DT; SMB=SKB/1024; print "NEW_USED: " $1 " KiB - OLD_USED: " $3 " KiB\nNEW_TIME: " $2 " s - OLD_TIME: " $4 " - DELTA_TIME: " DT "s\n---- SPEED: " SKB " KiB/sec, " SMB " MiB/sec ---" }'; echo $USEDkvk > /tmp/ddf-oldused; echo $TIMEkvk > /tmp/ddf-oldtime)
DU
#########################################
# du - new #
#########################################
# only the DU SPEED READING (manual) is new, the (auto/loop) works well so it didnt change. to find out whats new read below in the "du - old" section.
# DU SPEED READING (manual) - function that destroys itself (so it doesnt stay) - to run: "dusrm <folder>", by default "dusrm ." runs on current folder
function dusr { ( DIRd="."; FDd=`readlink -f "$DIRd"`; SUFd=`echo "$FDd" | sed 's/[^A-Za-z0-9._-]/-/g'`; TOLDf="/tmp/dusrt${SUFd}"; SOLDf="/tmp/dusrs${SUFd}"; TOLDd=`cat "${TOLDf}"`; SOLDd=`cat "${SOLDf}"`; SNOWd=$(du -s -B1 "${DIRd}" | tail -n 1 | awk '{print $1}'); TNOWd=`date +%s`; echo "* [`date`][`date +%s`/${INTERVAL123}s] du change '${DIRd}':"; echo "${SNOWd} ${SOLDd} ${TNOWd} ${TOLDd}" | awk '{printf " (%s-%s)bytes/(%s-%s)s = %sbytes/%ss = %s KiBps = %s MiBps\n", $1,$2,$3,$4,($1-$2),($3-$4),((($1-$2)/($3-$4))/1024),((($1-$2)/($3-$4))/1024/1024);}'; echo "${TNOWd}" > "${TOLDf}"; echo "${SNOWd}" > "${SOLDf}"; ); unset dusr; }; dusr .
######################
# DU SPEED READING (auto/loop) - measure du rate of change in FOLDER - set FOLDER and INTERVAL variable (seconds)
(FOLDER="."; INTERVAL=10; while true; do SNOW=$(du -s -B1 "${FOLDER}" | tail -n 1 | awk '{print $1}');TNOW=`date +%s`;echo "* [`date`][`date +%s`/${INTERVAL}s] du change in '${FOLDER}':";echo "${SNOW} ${SOLD} ${TNOW} ${TOLD}" | awk '{printf " (%s-%s)bytes/(%s-%s)s = %sbytes/%ss = %s KiBps = %s MiBps\n", $1,$2,$3,$4,($1-$2),($3-$4),((($1-$2)/($3-$4))/1024),((($1-$2)/($3-$4))/1024/1024);}'; TOLD=${TNOW};SOLD=${SNOW}; sleep $INTERVAL; done;)The end. By the way some of these scripts write some tmp files (containing size or time numbers which are recalled on the next run of the script to make the calculation).
DU – OLD – ARCHIVE (only manual version)
################################################################
# du - old - speed reading (manual) #
# - deprecated (kept for archive needs) #
################################################################
# old methods can only be run once per system because temp files that are created are the same. second rendition of du - old fixed that by allowing you set optional first $1 argument which will append text to the tempfiles and it then can differentiate the size and time values and thus provide correct results.
# new method can be ran as many times per system as you want. it will take the current folders full name. convert it to a proper filename (changes space and slashes to -, check out this article: http://www.infotinks.com/remove-invalid-characters-filenames/ ) and then that becomes the appended suffix to the temp files, thus it can always differentiate the size and time values per different folder that you run this in. Also I changed the new script to be a function that self destructs, so that you can specify which folder to run dusr at the right end of the oneliner, thus its easier to modify which folder to run as the cursor is automatically always placed on the right end after a copy paste (dusr is the name of this function, that does all of this). each time you run it just repeat the whole command (including the function statement each time, so that it can create the function, run it against your folder - default is . current folder - and then self destruct/ unset the function dusr). temp files are not deleted for obvious reasons so that next time the functon runs it has new and old values.
# new method above
# DU SPEED READING (manual) - pick FOLDER123 which you want to see the du (disk usage) rate of change (currently its set to current folder with the .)- edit FOLDER123 and run command and repeat commmand - rate of change measure from last time the command was run
(FOLDER123="."; SNOW123=$(du -s -B1 "${FOLDER123}" | tail -n 1 | awk '{print $1}');TNOW123=`date +%s`;echo "* [`date`][`date +%s`/${INTERVAL123}s] du change '${FOLDER123}':";echo "${SNOW123} `cat /tmp/dus-sold123` ${TNOW123} `cat /tmp/dus-told123`" | awk '{printf " (%s-%s)bytes/(%s-%s)s = %sbytes/%ss = %s KiBps = %s MiBps\n", $1,$2,$3,$4,($1-$2),($3-$4),((($1-$2)/($3-$4))/1024),((($1-$2)/($3-$4))/1024/1024);}'; echo ${TNOW123} > /tmp/dus-told123;echo ${SNOW123} > /tmp/dus-sold123;)
# -------------------------------- #
# you can run the above like this (assume above pasteable script is in /root/duspeed.sh)
# cd /folder/to/monitor/
# watch -n10 "/root/duspeed.sh"
# note you can only run this once per system because it uses the same tmp files to store information.
# if you want to run this on more than folder at the same time, we need to edit the script to produce different tmp files
# -------------------------------- #
# DU SPEED READING (manual) - pick FOLDER123 which you want to see the du (disk usage) rate of change - edit FOLDER123 and run command and repeat commmand - rate of change measure from last time the command was run. SUF can be left as $1.
(FOLDER123="."; SUF="$1"; SNOW123=$(du -s -B1 "${FOLDER123}" | tail -n 1 | awk '{print $1}');TNOW123=`date +%s`;echo "* [`date`][`date +%s`/${INTERVAL123}s] du change '${FOLDER123}':";echo "${SNOW123} `cat /tmp/dus-sold123` ${TNOW123} `cat /tmp/dus-told123`" | awk '{printf " (%s-%s)bytes/(%s-%s)s = %sbytes/%ss = %s KiBps = %s MiBps\n", $1,$2,$3,$4,($1-$2),($3-$4),((($1-$2)/($3-$4))/1024),((($1-$2)/($3-$4))/1024/1024);}'; echo ${TNOW123} > /tmp/dus-told123;echo ${SNOW123} > /tmp/dus-sold123;)
# -------------------------------- #
# now you can monitor several locations by either changing SUF to "1" for one folder and "2" for the other folder. Or since SUF is set the first argument of the script with $1 you can use it like so (assuming above copypasteable script is in /root/duspeed.sh):
# cd /monitor/first/folder/
# watch -n 10 "/root/duspeed.sh first"
# and in another shell (see: screen, terminal, putty, tmux, dtach)
# cd /monitor/second/folder/
# watch -n 10 "/root/duspeed.sh second"
…the end…