If your copying files using cp or rsync or whatever method. Sometimes you want to see the new files or changed files in the past N seconds.
You can do that like so
find /data -ls > /tmp/OLD # wait a few minutes find /data -ls /tmp/NEW
Now you can see the difference using “diff” or even “comm”
diff /tmp/OLD /tmp/NEW
Diff will show you the before and after. When looking at different files and what changed, we are just interested in seeing whats new, what changed. We are not interested in the old value (if you are then keep using “diff”). comm can help us isolate only whats new.
comm -13 /tmp/OLD /tmp/NEW
If you look at what comm does, it seperates whats unique to each file into different columns. So for example: all of the data unique to file1 (/tmp/OLD) will go in first column. all of the data unique to file2 (/tmp/NEW) will go in 2nd column. All of the data unique to both files (that appears in both /tmp/OLD and /tmp/NEW) will go in the 3rd column. using -1 (unique to file1),-2 (unique to file2), or -3 (unique to file3), we can suppress different outputs. For example from the above we only want to see whats new, so thats what is unique to /tmp/NEW (what lines only exist in /tmp/NEW in other words). So that means we only want the 2nd column. With comm, we dont tell it what we want, we tell it what we dont want. So we tell it to supress 1 and 3, leaving us with 2 – which is what is unique to file2 (/tmp/NEW).
Help output of comm might help
# comm --help
Usage: comm [OPTION]... FILE1 FILE2
Compare sorted files FILE1 and FILE2 line by line.
With no options, produce three-column output. Column one contains
lines unique to FILE1, column two contains lines unique to FILE2,
and column three contains lines common to both files.
-1 suppress column 1 (lines unique to FILE1)
-2 suppress column 2 (lines unique to FILE2)
-3 suppress column 3 (lines that appear in both files)
--check-order check that the input is correctly sorted, even
if all input lines are pairable
--nocheck-order do not check that the input is correctly sorted
--output-delimiter=STR separate columns with STR
--help display this help and exit
--version output version information and exit
Note, comparisons honor the rules specified by `LC_COLLATE'.
Examples:
comm -12 file1 file2 Print only lines present in both file1 and file2.
comm -3 file1 file2 Print lines in file1 not in file2, and vice versa.
Report comm bugs to bug-coreutils@gnu.org
GNU coreutils home page: <http://www.gnu.org/software/coreutils/>
General help using GNU software: <http://www.gnu.org/gethelp/>
Report comm translation bugs to <http://translationproject.org/team/>
For complete documentation, run: info coreutils 'comm invocation'
TIP: its best to sort the data for both files prior to running diff or comm. comm is more picky on it though.
find /data -ls | sort > /tmp/OLD # wait a few minutes find /data -ls | sort >/tmp/NEW # Now you can see the difference using "diff" or even "comm" diff /tmp/OLD /tmp/NEW comm -13 /tmp/OLD /tmp/NEW
Cool thats pretty cool. We can add some tweaks, like making ls output better.
find /data -type f -exec ls -lisa -B1 \; | sort > /tmp/OLD # wait a few minutes find /data -type f -exec ls -lisa -B1 \; | sort > /tmp/NEW # Now you can see the difference using "diff" or even "comm" diff /tmp/OLD /tmp/NEW comm -13 /tmp/OLD /tmp/NEW
ls -lisa -B1 shows entire contents of folders, so we dont want to run that on folders, we only run it on files. Either way we are only curious in seeing what files are new and get bigger. folders dont really make a difference unless there is something in them.
Now we can add loop
WATCH LOOP
watch is an interesting program that basically gives you the ability to do an infinite loop that nicely shows you data at the same time. I will use that to its advantage here.
This loops the following: ls all of the files with find and save that to /tmp/NEW, now diff /tmp/NEW and /tmp/OLD (first run of this fails because /tmp/OLD doesnt exist yet), now make a copy of /tmp/NEW to /tmp/OLD (move/rename would work as well), wait a few seconds, repeat… the second time around /tmp/OLD is the old find output, and /tmp/NEW is the new find output, so we can diff & comm them.
# simply fill out the DIR variable, or run as is to see difference in current working directory. these are one line commands, just copy paste them (need only edit the DIR var)
# comm file diff
DIR="."; watch -d -n10 "echo 'Changing files:'; find "${DIR}" -type f -exec ls -lisa -B1 {} \; | sort > /tmp/wNEW; comm -31 /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD;"
# diff file diff
DIR="."; watch -d -n10 "echo 'Changing files:'; find $DIR -type f -exec ls -lisa -B1 {} \; | sort > /tmp/wNEW; diff /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD;"
If ls freaks out you can just use the simpler find.
# simply fill out the DIR variable, or run as is to see difference in current working directory. these are one line commands, just copy paste them (need only edit the DIR var)
# comm file diff
DIR="."; watch -d -n10 "echo 'Changing files:'; find "${DIR}" -ls {} \; | sort > /tmp/wNEW; comm -31 /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD;"
# diff file diff
DIR="."; watch -d -n10 "echo 'Changing files:'; find $DIR -ls {} \; | sort > /tmp/wNEW; diff /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD;"
WHILE LOOP
Not all systems support “watch”. However all shells should have a while loop.
Here is one with the intricate better looking ls output
# comm (pick a line and use it, the second line clears the screen trying to imitate "watch" command, the first line makes the screen scroll)
DIR="."; INTS=10; while true; do echo 'Changing files:'; find "${DIR}" -type f -exec ls -lisa -B1 {} \; | sort > /tmp/wNEW; comm -31 /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD; sleep $INTS; done;
DIR="."; INTS=10; while true; do clear; echo 'Changing files:'; find "${DIR}" -type f -exec ls -lisa -B1 {} \; | sort > /tmp/wNEW; comm -31 /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD; sleep $INTS; done;
# diff (pick a line and use it, the second line clears the screen trying to imitate "watch" command, the first line makes the screen scroll)
DIR="."; INTS=10; while true; do echo 'Changing files:'; find "${DIR}" -type f -exec ls -lisa -B1 {} \; | sort > /tmp/wNEW; diff /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD; sleep $INTS; done;
DIR="."; INTS=10; while true; do clear; echo 'Changing files:'; find "${DIR}" -type f -exec ls -lisa -B1 {} \; | sort > /tmp/wNEW; diff /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD; sleep $INTS; done;
Here is one with the generic find included ls output
# comm (pick a line and use it, the second line clears the screen trying to imitate "watch" command, the first line makes the screen scroll)
DIR="."; INTS=10; while true; do echo 'Changing files:'; find "${DIR}" -ls | sort > /tmp/wNEW; comm -31 /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD; sleep $INTS; done;
DIR="."; INTS=10; while true; do clear; echo 'Changing files:'; find "${DIR}" -ls | sort > /tmp/wNEW; comm -31 /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD; sleep $INTS; done;
# diff (pick a line and use it, the second line clears the screen trying to imitate "watch" command, the first line makes the screen scroll)
DIR="."; INTS=10; while true; do echo 'Changing files:'; find "${DIR}" -ls | sort > /tmp/wNEW; diff /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD; sleep $INTS; done;
DIR="."; INTS=10; while true; do clear; echo 'Changing files:'; find "${DIR}" -ls | sort > /tmp/wNEW; diff /tmp/wOLD /tmp/wNEW; cp /tmp/wNEW /tmp/wOLD; sleep $INTS; done;
The end