My AXIS camera (AXIS M1011-W Network Camera) captures files into files that look like this:
i15-07-24_00-50-11-70.jpg i15-07-24_00-50-11-90.jpg i15-07-24_00-50-12-10.jpg i15-07-24_00-50-12-30.jpg
There are thousands of these images. The issue with lots of files of any type is that it takes forever to explore thru because Windows Explorer (or any file explorer for that matter), issue comes up with thosands of files and folders. I want them all organized into a date/hour folder format for easier and quicker viewing. So that I can easily navigate to the date and hour and look for the images.
# In the folder /data/Camera/one I have files like this: # - /data/Camera/one/i15-07-24_00-50-11-70.jpg # - /data/Camera/one/i15-07-24_00-50-11-90.jpg # - /data/Camera/one/i15-07-24_00-50-12-10.jpg # - /data/Camera/one/i15-07-24_00-50-12-30.jpg # I will run /root/scripts/orgcam.sh /data/Camera/one # And they get moved to # - /data/Camera/one/2015-07-24/00/i15-07-24_00-50-11-70.jpg # - /data/Camera/one/2015-07-24/00/i15-07-24_00-50-11-90.jpg # - /data/Camera/one/2015-07-24/00/i15-07-24_00-50-12-10.jpg # - /data/Camera/one/2015-07-24/00/i15-07-24_00-50-12-30.jpg
The settings that I have for the events that get recorded are setup like this (the important setting that changes the file name is the Base file name which I have set to i.jpg, thats why my files start with i. Then the Add date/time suffix option need to be checked so that we get the proper date format of <base file><2 digit year>-<2 digit month>-<2 digit day>_<2 digit hour>-<2 digit minute>-<2 digit second>-<hundredth of second>.<extension of base file, which is jpg>). You get to this setup menu after you your servers (where you want the images saved to), when you setup the event type (the type of events that you want recorded & you select which server they are saved to & using which format like FTP & the file format – base file name and whether or not you want to add the date/time suffix). This is located in the WEB GUI under Setup in the Events tab, Event Types sub tab, when you create or modify an event.
Please read the comments in this script to see how to use it.
NOTE: this can be used to organize any files that have the format xYY-dd-mm_HH-MM-SSy. Where ‘x’ is any number of ascii chars that are not numbers – ‘x’ could be blank (in example below x is the character ‘i’, ‘YY’ is 2 digit year, ‘dd’ is 2 digit day, ‘mm’ is 2 digit month, ‘HH’ is 2 digit hour, ‘MM’ is 2 digit minute, ‘SS’ is 2 digit second, ‘y’ is the suffix which usually contains another time unit and an extension (.jpeg). At a minimum you are required to contain this xYY-dd-mm_HH. Files that will work (only need the minimum): i15-07-24_00-50-11-70.jpg, i15-07-24_00-50-11-70, da15-07-24_00-50-11, abc15-07-24_00-1.txt, 15-07-24_00-1.txt”
The script:
#!/bin/bash
# script name: orgcam.sh
# create date: 2015-09-30
# author: infotinks
#
# What this does: Organize AXIS camera captures images, or other files that follow the same date format, by moving them into a dated folder (YYYY-mm-dd) with the hour subfolder (HH). "file" -> "YYYY-mm-dd/HH/file". This is good if you have a folder with thousands of images that have a date that you want to arrange.
#
# Example: Looking at the settings of AXIS, when it saves the images from a capture, you have to set the basename & also if you want to "Add date/time suffix" (you need this option enabled for this script to work or to even make sense, if you dont use "Add date/time suffix" there is no point in using this script to organize the files). In this example I have the following:
# - BASE FILENAME: i.jpg
# - I have also checked "Add date/time suffix"
# - This gives us files that look like this: i15-07-24_00-50-09-70.jpg
# - In the folder /data/Camera/one I have files like this:
# - /data/Camera/one/i15-07-24_00-50-11-70.jpg
# - /data/Camera/one/i15-07-24_00-50-11-90.jpg
# - /data/Camera/one/i15-07-24_00-50-12-10.jpg
# - /data/Camera/one/i15-07-24_00-50-12-30.jpg
# - I will run /root/scripts/orgcam.sh /data/Camera/one
# - And they get moved to
# - /data/Camera/one/2015-07-24/00/i15-07-24_00-50-11-70.jpg
# - /data/Camera/one/2015-07-24/00/i15-07-24_00-50-11-90.jpg
# - /data/Camera/one/2015-07-24/00/i15-07-24_00-50-12-10.jpg
# - /data/Camera/one/2015-07-24/00/i15-07-24_00-50-12-30.jpg
#
# point at the directory where the camera images are:
# /root/scripts/orgcam.sh /data/Camera/one
(
if [ $# -eq 0 ]; then
echo "* ERROR: no arguments supplied"
echo "- Usage: $0 <folder where files>"
echo "- The files should be images that are formated like so: xYY-dd-mm_HH-MM-SSy"
echo "- Where 'x' is any number of ascii chars that are not numbers - 'x' could be blank (in example below x is the character 'i', 'YY' is 2 digit year, 'dd' is 2 digit day, 'mm' is 2 digit month, 'HH' is 2 digit hour, 'MM' is 2 digit minute, 'SS' is 2 digit second, 'y' is the suffix which usually contains another time unit and an extension (.jpeg). At a minimum you are required to contain this xYY-dd-mm_HH. Files that will work (only need the minimum): i15-07-24_00-50-11-70.jpg, i15-07-24_00-50-11-70, da15-07-24_00-50-11, abc15-07-24_00-1.txt, 15-07-24_00-1.txt"
echo "- Example: $0 /data/Camera/backyard"
echo "- The above example will do the following:"
echo "- /data/Camera/backyard/i15-07-24_00-50-11-70.jpg --> /data/Camera/backyard/2015-07-24/00"
echo "- /data/Camera/backyard/i15-07-24_00-50-11-90.jpg --> /data/Camera/backyard/2015-07-24/00"
echo "- /data/Camera/backyard/i15-07-24_00-50-12-10.jpg --> /data/Camera/backyard/2015-07-24/00"
echo "- /data/Camera/backyard/i15-07-24_00-50-12-30.jpg --> /data/Camera/backyard/2015-07-24/00"
exit 1
fi
DIR=${1} # the first argument which is the date
U=/dev/null # we will be outputting lots of errors into the garabage /dev/null. $U is shorter to type then /dev/null
FDIR=$(readlink -f "${DIR}" 2> $U)
IFS=$'\n' # this is the for loop delimiter, so that it doesnt delimit on space, but only on newlines, so that files with names or dirs that have spaces get processes correctly.
N_INVDATE=0; # setting counter var to 0, how many files dont have a valid date
N_TOTAL_MV=0; # setting counter var to 0, total number of moves attempted
N_GOOD_MV=0; # setting counter var to 0, total number of successful move operations
N_BAD_MV=0; # setting counter var to 0, total number of failed moves (could be permission related, or folder got deleted that they were moving to)
N_TOTAL=0; # setting counter var to 0, total number of files processes (files with good format and bad format)
N_FOLDER_CREATE_GOOD=0; # setting counter var to 0, total number of folders created successfully
N_FOLDER_CREATE_FAIL=0; # setting counter var to 0, total number of folders failed to create (could be permissions related)
N_FOLDER_CREATE_TOTAL=0; # setting counter var to 0, total number of folders that asked to create (failed + success)
echo "Organizing files with format [char]YY-MM-DD_HH-mm-ss"
echo "Asked to look in: ${DIR}"
echo "Looking in: ${FDIR}"
if [ ! -d "${FDIR}" ]; then
echo "* ERROR: folder doesnt exist"
exit 1;
fi
echo "##########################################################"
T0=`date +%s` # start date in epoch seconds (seconds since 1970-01-01 essentially)
for i in $(find "${FDIR}" -maxdepth 1 -type f 2> $U); do
N_TOTAL=$((N_TOTAL+1))
FNAME=$(basename "${i}" 2> $U) # i15-07-24_00-50-09-70.jpg
FDATE=$(echo $FNAME 2> $U | sed 's/^[a-z]*\([0-9]*-[0-9]*-[0-9]*\)_.*$/\1/' 2> $U) # 15-07-24
FYEAR2DIGIT=$(echo $FDATE 2> $U | cut -f1 -d"-" 2> $U) # 15
FYEAR4DIGIT=$(date --date="$FYEAR2DIGIT-01-01" +%Y 2> $U) # 2015, by asking whats 4 digit year of "2015-01-01"
FMONTH=$(echo $FDATE 2> $U | cut -f2 -d"-" 2> $U) # 07
FDAY=$(echo $FDATE 2> $U | cut -f3 -d"-" 2> $U) # 24
NDATE="${FYEAR4DIGIT}-${FMONTH}-${FDAY}" # 2015-07-24
FHOUR=$(echo ${FNAME} 2> $U | cut -d"_" -f2 2> $U | cut -d"-" -f1 | sed -r 's/([^0-9]*([0-9]*)){1}.*/\2/' 2> $U) # 00 - that last sed grabs the first number there to avoid errors like this: "no12-11-29_23abc.txt" which will get an FHOUR of 23abc.txt. That sed statement makes FHOUR just 23 so it works. These errors will not happen as i only expect this script to be run on AXIS file outputs.
NEWDIR="${FDIR}/${NDATE}/${FHOUR}"
VALIDATION_DATE="${NDATE} ${FHOUR}"
if date --date=${VALIDATION_DATE} > $U 2> $U; then # check if camera file by seeing if it has a valid extracted date. if this was a none camera file VALIDATION_DATE would be something that date wouldnt understand as a date
if [ ! -d "$NEWDIR" ]; then
N_FOLDER_CREATE_TOTAL=$((N_FOLDER_CREATE_TOTAL+1))
echo -n "* creating folder ${N_FOLDER_CREATE_TOTAL} ${NEWDIR}: "
if ! mkdir -p "${NEWDIR}" > $U 2> $U; then
TMP_EXIT_STATUS=$? # hold exit status of mkdir command
N_FOLDER_CREATE_FAIL=$((N_FOLDER_CREATE_FAIL+1))
echo "* ERROR: failed creating folder #${N_FOLDER_CREATE_FAIL}, mkdir exit status: ${TMP_EXIT_STATUS}"
continue;
else
N_FOLDER_CREATE_GOOD=$((N_FOLDER_CREATE_GOOD+1))
echo "SUCCESS, folder created #${N_FOLDER_CREATE_GOOD}"
fi;
fi
N_TOTAL_MV=$((N_TOTAL_MV+1))
echo -n "-- file #${N_TOTAL}, move #${N_TOTAL_MV}: mv \"${i}\" \"${NEWDIR}\": "
if mv "${i}" "${NEWDIR}" > $U 2> $U; then # try to move file, if fails log
N_GOOD_MV=$((N_GOOD_MV+1))
echo "SUCCESS, file moved #${N_GOOD_MV}"
else
TMP_EXIT_STATUS=$? # hold exit status of move command
N_BAD_MV=$((N_BAD_MV+1))
echo "ERROR: failed move #${N_BAD_MV}, mv exit status: ${TMP_EXIT_STATUS}"
fi
else
N_INVDATE=$((N_INVDATE+1))
echo "* ERROR: invalid date #${N_INVDATE}, \"$i\", extracted date ${VALIDATION_DATE} (correct format should be YYYY-MM-DD HH)"
fi
done;
T1=`date +%s` # finish date in epoch seconds (seconds since 1970-01-01 essentially)
TDIFF=$((T1-T0)); # subtracting start from finish date in seconds to get total run time
FPS=$(echo "N_TOTAL $TDIFF" | awk '{print $1/$2}') # files processed / second (all files, moved & not moved, and invalid date)
echo "##########################################################"
echo "stats:"
echo "* startime: $T0, `date --date=@$T0 +%Y-%m-%d_%H-%M-%S`; endtime: $T1, `date --date=@$T1 +%Y-%m-%d_%H-%M-%S`"
echo "* total run time (total m:s, or total s): $(($TDIFF / 60)):$(($TDIFF % 60)), or ${TDIFF}"
echo "* files processed per second (moved & not moved): $FPS"
echo "* files processed:"
echo "* mkdirs: ${N_FOLDER_CREATE_TOTAL} total, ${N_FOLDER_CREATE_GOOD} good, fail ${N_FOLDER_CREATE_FAIL} failed"
echo "* invalid date files (untouched files): ${N_INVDATE}"
echo "* mvs: ${N_TOTAL} total, ${N_GOOD_MV} good, ${N_BAD_MV} failed"
exit 0
)
SIDENOTE: you can rerun the scripts it will not reorganize files that have already been organized. It only looks at the files in the root of the directory you point orgcam.sh to.
SIDENOTE: If you have your different cameras or events saving to different folders like this /data/Camera/cam1,/data/Camera/cam2 and etc… Inside the cam1 and cam2 files is where the images are. You can use a wrapper script that basically just runs orgcam.sh against the first layer of subfolder. so it will run orgcam.sh first at cam1 and then at cam2 (and the end result organized files will be in their respectable cam1 or cam2 folder -in other words it will not mix cam2 files into cam1 folders). Then you can set that script to run daily with your cron job.
#!/bin/bash
# file: /root/scripts/cron-orgcam-here.sh
# run orgcam on every subfolder in /data/Camera
# set to run daily with a cron job. Example: Run "crontab -e"
# Add this line @ the bottom:
# 12 10 * * * /root/scripts/cron-orgcam-here.sh
# This runs cron-orgcam-here daily at 12:10 am. Make sure to only not write the hash pound or else that would be a comment.
(D="/data/Camera";
O="/root/scripts/orgcam.sh"
IFS=$'\n';
DS=`find "${D}" -maxdepth 1 -type d | grep -v "^$D$"`
for i in $DS; do
echo "Running $O on $i"
$O $i
done;)
The End.