A Snoopy How To – plus a mini explanation to linux libraries

Snoopy works for linux (it might also work for unix, im not sure). Here is watch_execve which works for FreeBSD: http://www.infotinks.com/snoppy-freebsd-watch_execve-watch-command-executions-freebsd/

---

 

Snoopy logs all commands & programs that are ran by any user (or program) to a log file. This is great for auditing and finding out things. For example: you have a NAS and you run a Scrub operation from its GUI and your curious as to what commands are actually ran with that operation. It will show you exactly what commands are ran. If it doesnt show anything, most likely all of the commands were c/c++ system calls. Note that Snoopy cant pick up system and library calls via binary files, it can only pick up unix programs that are ran and their arguments (things you could run in shell like bash). So if a unix read function and write function was used to copy a file, then snoopy wont show you that, but it will show you if someone uses “cat” or “cp”.

---

 

* UPDATE 2015-10-21: This gets updated often. So to get the latest install instructions. Just go here: https://github.com/a2o/snoopy. Latest instructions for install are this:

First run this to get your dependencies up and running:

# install these dependencies
# run these one by one (autoconf and git might be optional)
apt-get install build-essential
apt-get install autoconf
apt-get install git

To install latest STABLE version:

rm -f snoopy-install.sh && wget --no-check-certificate -q -O snoopy-install.sh https://github.com/a2o/snoopy/raw/install/doc/install/bin/snoopy-install.sh && chmod 755 snoopy-install.sh && ./snoopy-install.sh stable

To install latest DEV version:

rm -f snoopy-install.sh && wget --no-check-certificate -q -O snoopy-install.sh https://github.com/a2o/snoopy/raw/install/doc/install/bin/snoopy-install.sh && chmod 755 snoopy-install.sh && ./snoopy-install.sh git-master

NOTE: I added in –no-check-certificate so that any unix system can wget it even it doesnt have the latest ca-certs installed

NOTE: on a debian system to install the dependencies & install snoopy stable run this  apt-get update && apt-get install build-essential -y && apt-get install autoconf -y && rm -f snoopy-install.sh && wget –no-check-certificate -q -O snoopy-install.sh https://github.com/a2o/snoopy/raw/install/doc/install/bin/snoopy-install.sh && chmod 755 snoopy-install.sh && ./snoopy-install.sh stable  and if you wanted snoopy dev version run this apt-get update && apt-get install build-essential -y && apt-get install autoconf -y && rm -f snoopy-install.sh && wget –no-check-certificate -q -O snoopy-install.sh https://github.com/a2o/snoopy/raw/install/doc/install/bin/snoopy-install.sh && chmod 755 snoopy-install.sh && ./snoopy-install.sh git-master

---

*  UPDATE 2014-10-24: There is now a very easy way to install snoopy and now you can activate and deactivate snoopy with a command (sweet!). Easier then all of the other ways.

# ---- Snoopy Easy Install: ---- #
wget -q -O - https://github.com/a2o/snoopy/raw/master/bin/snoopy-install.sh | sh

# If it does nothing (it might be because you have missing root certs on your linux box):
wget --no-check-certificate -q -O - https://github.com/a2o/snoopy/raw/master/bin/snoopy-install.sh | sh

# if install fails, try to make sure you preinstall these and then try the installer again:
apt-get update
apt-get install build-essential
apt-get install autoconf
apt-get install git


# ---- check out install source code ---- #

# if your curious to check out the code
cd ~
wget https://github.com/a2o/snoopy/raw/master/bin/snoopy-install.sh
# and if you have cert troubles wget it like so:
wget --no-check-certificate  https://github.com/a2o/snoopy/raw/master/bin/snoopy-install.sh


# ---- activating and deactivating snoopy ---- #

# to activate snoopy: snoopy-enable
# to deactive snoopy: snoopy-disable

---

 

* UPDATE 2014-10-24: word from maintainer of snoopy code – new code location: https://github.com/a2o/snoopy

git clone https://github.com/a2o/snoopy

If fail with SSL error look at this article SSL ERROR

GIT_SSL_NO_VERIFY=true git clone https://github.com/a2o/snoopy

Links below will fail, the above links will work. Still need to update scripts below to include above links.

---

 

So What is Snoopy?

Snoopy allows for command monitoring thru syslog (essentially every command that has been launched on the system)

The solution to my problem, see here is my problem before I found snoopy: HERE!

Snoopy, a very small and hidden program

Much smarter then my scripts in the above link

HOW IS THIS USEFUL FOR TESTING

If your using an application that has a web GUI(or QT or anyother GUI) that launches commands on the back-end of the server. Such as via tomcat or apache  (cgi) or python webserver or anythoer. First enable snoopy with snoopy-enable, then make a fake log entry logger “aaaaaaaaaaaaaaaaaaaaaaaaaaaa”, then run the gui command, and after the gui command is complete run logger “aaaaaaaaaaaaaaaaaaaaaaaaaaaa” again.

The check out your logs and you will see what the GUI did to the backend, they will be inbetween the aaaaaaaaaaaaaaaa command. Tthis will not show any C, or C++ commands, or bash embedded commands such as bg,fg,sleep, etc unless your using gnu sleep, and gnu bg, and gnu fg – not even sure if gnu bg and gnu fg exist – but for example ls can be both bash embedded or there is a version of ls that is gnu, you would see the gnu command. Also another caveat is snoopy doesnt show the pipe symbols, so when your looking at the output you need to make educated guesses to where you think they are piped. So If see for example a ps command and then next line is a grep and then an awk. Well then I can assume that ps | grep | awk was how that command went.

---

 

Note that now you cant install snoopy using the method below. If your curious about the installation method, just check out the snoopy-install.sh (see above on how to wget it) script. Even though the install steps below will not apply, please continue on reading and see how snoopy works at the core, you will also learn more about “librarys” – that stuff is key in linux.

Old Install Method

Quick Summary:

 

Made quick summary because this article is so long, didnt want it to bee too tl;dr.

############################
# Summary of full install: #
############################

# To Install:
apt-get install build-essential
apt-get install autoconf
apt-get install git

apt-get install git
mkdir -p /root/src
cd /root/src
git clone git://git.code.sf.net/p/snoopylogger/code snoopylogger-code
cd snoopylogger-code/
# or the folder you need to cd in could be called "code" instead of "snoopylogger-code" so then do this, if above cd fails:
# cd code/
# or just do this to find out what git clone downloaded
# and go into that folder:
# ls -lisah

# once in that folder
vi snoopy.h
# Change the line that says:
#define SNOOPY_MAX_ARG_LENGTH 4096
# To:
#define SNOOPY_MAX_ARG_LENGTH 12288

# Or with 1 liner:
sed -i "s/4096/12288/g" snoopy.h

autoheader
autoconf
./configure

make

#####################################################
# (skip this section if running linux)
# FIX FOR OPENSOLARIS: IF MAKE FAILS IN OPENSOLARIS OR NEXENTAOS
# THE PROBLEM: running "make" gives this error (or something similar to it):
# snoopy.c:136:29: error: 'LOG_AUTHPRIV' undeclared (first use in this function)
# THE REASON AND GENERAL INFO: Its because syslogging is not done by LOG_AUTHPRIV but by LOG_AUTH
# the fix is to simply edit "config.h" file (do this after running the "./configure" command and before running the "make" command.
# If you already ran "make" its okay, just do this fix and run "make" again)
# HERE IS THE FIX: "vi config.h" then find "LOG_AUTHPRIV" (it should only appear once) and change it to "LOG_AUTH", then save and exit, then run "make" and continue with rest of guide.
# So basically, the line in "config.h" should say this:
#define SNOOPY_SYSLOG_FACILITY LOG_AUTHPRIV
# And you should change it:
#define SNOOPY_SYSLOG_FACILITY LOG_AUTH
######################################################

make install
make enable

# If "make enable" failed:
# --method1--
# Find where installed (most likely it installed to /usr/local/lib with its final name being /usr/local/lib/snoopy.so):
find / -name "snoopy.so"
# Use the answer of that, must use full absolute path with directory name
echo "/usr/local/lib/snoopy.so" >> /etc/ld.so.preload
# --method2--
# when make enable fails, you will see that it tried to run:
# ./enable.sh /usr/local/lib
# where /usr/local/lib might be different for your system (but doubt it, unless you messed with ./configure options - note messing with LOG_AUTH in solaris fix wouldnt affect this)
# try manually running it, and that might do it, confirm that it wrote the file to /etc/ld.so.preload: 
# cat /etc/ld.so.preload
# if it didnt and that file is empty or doesnt exit just do method1

The end, now it should work

###########
# To Use: #
###########

# If syslog (works in systemd and sysvinit systems):
cd /var/log
grep -r "snoopy" *
# For me I saw all of the output goes to auth.log
tail -f /var/log/auth.log | grep snoopy

# If systemd:
journalctl -fa | grep snoopy

# to get just the command line parts without all of the extra meta-info (like username and stuff) look at article below

#################
# To Uninstall: #
#################

# 1. delete snoopy.so
rm /usr/local/lib/snoopy.so

# 2. remove the reference to /usr/local/lib from ld.so.preload
vi /etc/ld.so.preload
# scroll down to that line and hit "dd" twice followed by ":wq!" to save, all those without typing the double quotes/
# Or use this oneliner to do the same thing:
sed -i "/snoopy.so/d" /etc/ld.so.preload

# 3. Unset LD_PRELOAD, LD_PRELOAD_32 and LD_PRELOAD_64 system variables, but most likely you wont need to do that, as they are not set in my system.

 

Here is how snoopy works:

A program needed to be inveted that recorded when other programs run, but how do you do that? well make sure it ALWAYS runs before your intended program runs, well how do you do that? by preloading libraries (libraries are mini programs, but in reality are tool belts that can do things, so yah they are like mini programs, that can either be compiled and shoved into the final program/executable of your choice, or shared with other final programs/executables of your choice – more on this below)

Whenever a program runs in linux, it checks the libraries which the program needs and loads them, it also checks the file /etc/ld.so.preload for any libraries and preloads them. NOTE: ld.so.preload is a text file that is line delimited, every new line is a library that should be loaded before a program is ran.

A good article on /etc/ld.so.preload: http://minipli.wordpress.com/2009/07/17/ld_preload-vs-etcld-so-preload/

Anything in /etc/ld.so.preload gets preloaded for EVERY program. So the library which gets called is the snoopy library/program. That library intercepts the command name and who ran it and appends that information to syslog.

The result is command monitoring just as I wanted it

Pre install and download:

To install it, first prepare your compiling environment (you will need to have things installed like make and gcc, hopefully you already have them):

 

apt-get update

# to get most of the build tools (hopefully these 2 commands get everything for you)
apt-get install build-essential

# to get the extra autoconf and autoheader tool (these tools generate a configure script for us. this is not in build-essentials because usually when compiling source, autoconf and autoheader have been done and you already have the ./configure script)
apt-get install autoconf

 

apt-get install git
mkdir -p /root/src
cd /root/src
git clone git://git.code.sf.net/p/snoopylogger/code snoopylogger-code
cd snoopylogger-code/
# if cd into snoopylogger-code fails then cd into code
# cd code/
# if both fail, find out the name of the folder git dumped and cd into it: ls -lisah

NOTE: when running git clone it will download the source code into a folder name that they chose, in this case they chose code. You can just run ls -lisah to see whats the name of the folder that git clone downloaded

OPTIONAL STEP BEFORE INSTALLING: Allow for snoopy to record bigger arguments in commands. change the snoopy.h file so that the snoopy installation allows for recording bigger arguments:

vi snoopy.h

Find this line at the top:

#define SNOOPY_MAX_ARG_LENGTH 4096

change to 3 times the size (thanks to video by Kelvin Tan Thiam Teck at this link http://www.youtube.com/watch?v=Lts34JkmwuU)

#define SNOOPY_MAX_ARG_LENGTH 12288

 

Or with a oneliner using sed you can replace it like this:

sed -i "s/4096/12288/g" snoopy.h

 

We basically set the global variable which sets the max size of arguments snoopy records.

Back to install:

BEFORE CONFIGURE (you will notice that this git clone doesnt have the configure script, so you can make it with autoheader and autoconf – dont worry I got this from the README file which tells you exactly what to do):

autoheader
autoconf
./configure

make
make install

NOTE: if running solaris or opensolaris, the make should fail, you will need to run a fix, that fix is detailed out at top in the summary

Now the software has put its library in the OS library directories. Now its up to you to enable the library (enable in snoopys terms meaning, tell /etc/ld.so.preload about this snoopy lib, so that the snoopy lib gets loaded before every program). Test if for your self do a test of: # cat /etc/ld.so.preload, # make enable, and cat it again # cat /etc/ld.so.preload

make enable

ONLY READ THIS NEXT PARAGRAPH IF “make enable” FAILED: if the make enable fails (confirm it failed by running cat /etc/ld.so.preload , if you dont see anything there most likely it failed, also if you dont see the path where snoopy.so was isntalled to then it also failed), just run it manually, you will see what it tried to run its something like ./enable.sh /usr/local/lib . Dont take my word for it, on your system it could be something other then /usr/local/lib. Its just going to the folder where snoopy.so was installed. You can find the full path and just truncate (or keep everything) to the left of the last /, using this command: find / -name “snoopy.so” . After getting the folder just run the above command ./enable.sh <folder where snoopy.so is at> , replace <folder where snoopy.so is at> with the actual folder, not including the angle brackets (sorry I have to be so descriptive, thats why all my stuff is so tl;dr). Also instead of running ./enable.sh, you can just manually do what it was going to do. Just edit or create (if its missing) the file called /etc/ld.so.preload. Here is the command to manually do what make enable would do, if your snoopy.so is in /usr/local/lib: echo “/usr/local/lib/snoopy.so” >> /etc/ld.so.preload .  Obviously replace /usr/local/lib/snoopy.so with the full-file name (folder name included)  – aka: absolute path – – where snoopy.so installed for you.

Thats it its now enabled. Skip to section “how to use” if you dont want to get in the details of how this snoopy is installed (i covered some of it above already).

So you can see that “make enable” writes to “/etc/ld.so.preload” one line with the location of the snoopy lib.

My ld.so.preload looks like this (and yours will probably too because the source code compilation with ‘configure’ and ‘make’ and ‘make install’ put it there):

# cat /etc/ld.so.preload
/usr/local/lib/snoopy.so

To test to see that /etc/ld.so.preload gets checked out before any program try this: “# apt-get install strace” now do an strace (monitors lib and syscalls) on any program “# strace touch test123” “strace rm test123”. Notice that in the strace you see

access("/etc/ld.so.preload", R_OK) = 0
open("/etc/ld.so.preload", O_RDONLY) = 5

Then later you see:

open("/usr/local/lib/snoopy.so", O_RDONLY) = 5

 

How to use:

Now that snoopy is enabled your done, you can now just delete the source directory, and begin running commands.

To see the commands, we need to look thru where syslog saves its output

cd /var/log

grep "snoopy" *

or if you have nested log folders do a recursive search

grep -r "snoopy" *

Notice where all of the snoopy output goes

For me all of the latest commands were being written to auth.log

So for me to follow the latest commands:

tail -f /var/log/auth.log | grep snoopy

NOTE: example output can be seen below

More in depth – How to use:

Sorry this whole next part is in code blocks, everything I wanted to say is in comments after # lines.

 

################################
# MONITORING IN SYSLOG SYSTEMS #
################################

# note this also works in systemd

# In syslog systems (also works in systemd systems): 

cd /var/log
grep -r snoopy *

# see which file all the snoopy commands go to, in this case its auth.log

# after finding out my logs go to auth.log (so below replace auth.log) with whatever
tail -f auth.log

# to grab just the snoopy stuff
tail -f auth.log | grep snoopy

# Notice the output is like:
# Feb 24 16:48:10 server123 snoopy[27320]: [uid:0 sid:1670 tty: cwd:/ filename:/bin/sleep]: sleep 10
# notice everything after the last ]: is the command, so lets select that out

# to grab just the command information
tail -f auth.log | grep snoopy | sed -n -e 's/^.*]: //p'

###################################
# MONITORING WITH SYSTEMD SYSTEMS #
###################################

# journalctl is used in systemd to monitor logs, with -fa we can follow all logs, the -f is for follow, -a is for all logs, or else they are truncated

# In systemd systems:
# We can follow the logs:
journalctl -fa

# We can follow the snoopy output:
journalctl -fa | grep snoopy

# Notice the output is like:
# Feb 24 16:48:10 server123 snoopy[27320]: [uid:0 sid:1670 tty: cwd:/ filename:/bin/sleep]: sleep 10
# notice everything after the last ]: is the command, so lets select that out

# We can follow the snoopy output after the command stuff:
journalctl -fa | grep snoopy | sed -n -e 's/^.*]: //p'

###################################
# BONUS INFO: Whats the sed doing #
###################################

# From this site: http://unix.stackexchange.com/questions/24140/return-only-the-portion-of-a-line-after-a-matching-pattern
# thanks to username: Gilles

# The canonical tool for that would be sed.

sed -n -e 's/^.*stalled: //p'

# Detailed explanation:

# -n means not to print anything by default.
# -e is followed by a sed command.
# s is the pattern replacement command.
# The regular expression ^.*stalled: matches the pattern you're looking for, plus any preceding text (.* meaning any text, with an initial ^ to say that the match begins at the beginning of the line). Note that if stalled: occurs several times on the line, this will match the last occurrence.
# The match, i.e. everything on the line up to stalled:, is replaced by the empty string (i.e. deleted).
# The final p means to print the transformed line.

# in my case I did:
# sed -n -e 's/^.*]: //p' 
# I thought I would have to do it twice because if you look we have the occurance of ]: twice, but doing it only once worked (it grabbed the last one - i thought I would need to grab the first one and everything after, and repeat to actually get the result that i wanted)

 

If not working:

Restart your shell or reboot:

If you cant see anything being recorded.

if your in ssh, restart ssh, dont worry you connection should maintain if your ssh is smart
if sysvinit/init system:

/etc/init.d/ssh restart

if systemctl:

systemctl restart ssh

or

systemctl restart ssh.service

or

Simply log out and log back in

Also try rebooting the system, thats the perfect test. The test even more perfect then a reboot? Reboot again, then if the program works without problems after 2 reboots your sure as @#$@ know it works.

Make sure programs link to snoopy.so library:

Mini intro to libraries in linux:

static libraries are extra tools for your program to use which end with .a extension and they are compiled into the program making them bigger (but the tools / the library always comes embedded into the final program thus you dont need to install any shared libs as a prequiste.) PRO: comes with app, no preqs, CON: bigger footprint. The library and the program sit together as 1 file in the end (you still have the original shared lib file sitting in your lib folder somewhere, so that it can be used by other program)

dynamic libraries are with the extension .so (plus maybe extra numbers after like .so.1) then are not compiled into the program but they are linked to the program. if 10 programs are using the same 11K library, that library will only take up 11K of memory. However if it was a shared library then the footprint would be 11K for every lib times 10 apps using it (using a seperate version of it, that sits compiled into each program) so it would be a 110k total foot print (10 times bigger then if it were a dynamic library). the disadvantage of course. PRO: smaller footprint and shared, CON: need to have it in the system as an extra file

ldd command can be used to see which librarys (its like windows dlls, they are not programs but contain extra functions and methods that a program can use, “my analogy to what libraries and dll files  are like:hey you dont want to retype an ‘cos’ and ‘sin’ function everytime you write a program that uses basic math) are dynamically linked(also known as shared librarys) thus a single library can be used by multiple running programs at once (so the same lib doesn’t have to load as many times as there are programs using it, this saves memory). A static library compiles the librarys into the executable (so they dont require library prerequisites, as they are compiled into the program). ldd shows you if a program is dynmically links/shared by showing which libraries it uses (you can see snoopy being called right after the vdso library-sidenote: vdso allows for faster cpu interrupts by using SYSENTER and SYSEXIT instead of intruppts – this was important when cpus became faster and intruppts bottleneck performance), ldd shows you if its a static library by telling you that the program has no dynamic links (no shared libraries associated to it). So we see he snoopy although is dynamically linked to ln and touch will show when those commands run. Also even though we see that zcat is static and doesn’t mention snoopy, it still runs snoopy so zcat is recorded into syslog, but how its not linked? well my guess is ld.so.preloads work by linking thier uses on top of programs whether they are static linked or shared link, that way you ensure consitency with all apps running libs in ld.so.preloads. BONUS TIP: if a program is compiled with static, then all of the libraries are in the final executable file so the program will be bigger, but can run without prerequesite libraries laying around in your system

# SYNTAX:
ldd <full program path>

# DYNAMICALLY LINKED (shows snoopy)
ldd `which ln`
        linux-vdso.so.1 =>  (0x00007fff6899e000)
        /usr/local/lib/snoopy.so (0x00007f7d5e7a3000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7d5e400000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7d5e1fc000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f7d5e9a8000)

# DYNAMICALLY LINKED (shows snoopy)
ldd `which touch`
        linux-vdso.so.1 =>  (0x00007fffa47ff000)
        /usr/local/lib/snoopy.so (0x00007f9439c5f000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f9439a3e000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f94396b4000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f94394b0000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f9439294000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f9439e64000)

# STATIC LIBRARY (doesnt show snoopy but dont worry it still runs and records the command with snoopy to syslog)
ldd `which zcat`
        not a dynamic executable

 OUTPUT FROM BOTH PROOVING THAT BOTH TYPES OF DYNMICALLY AND STATIC LINKED APPS GET RECORDED:

Example output of snoopy logging in auth

# cat /var/log/auth.log | grep snoopy
Feb 22 22:36:55 debikos71 snoopy[2798]: [uid:0 sid:23025 tty:/dev/pts/28 cwd:/var/log filename:/usr/bin/touch]: touch test123
Feb 23 07:07:43 debikos71 snoopy[12187]: [uid:0 sid:7117 tty:/dev/pts/0 cwd:/var/log filename:/bin/zcat]: zcat /root/ncdu-scripts/results/ncdu-full-e1393134884-d2014-02-22-t21-54-44.gz

NOTE: for me and my default rsyslog/syslong config, all of the snoopy output goes to auth.log

To uninstall:

1. just remove the reference line in /etc/ld.so.preload

sed -i '/snoopy/d' /etc/ld.so.preload<span style="background-color: #ffffff; font-family: Lato, sans-serif; font-size: 16px; line-height: 1.5;"> </span>

Or manually go in

vi /etc/ld.so.preload

Now delete the line that mentions snoopy.so

2. Now delete the /usr/local/lib/snoopy.so

If yours is else where and you cant remember where

find / -iname "snoopy.so"

To delete what it find

find / -iname "snoopy.so" -exec rm -i {} \;

or easier but more confidently:

find / -iname "snoopy.so" -delete

3. Also unset any environmental variable that references snoopy (LD_PRELOAD, LD_PRELOAD_32 and LD_PRELOAD_64).

Caveats and limitations of snoopy

1- I found one problem/limitation with snoopy is that it wont capture a full command if your arguments are bigger then 4096 characters (or if you changed the gloval variable in snoopy.h before compilation then you can increase it to like 3 times the amount of charcters – so that limitation isnt really big)

2- The big limitation is when you pipe commands, it captures each command and its arguments seperately

command1 | command2 | command3

each command shows up seperately in the final snoopy syslog capture.

# I ran this chain of commands piped together to show you

 cat /var/log/messages | grep -i systemd | grep ssh
# I had this in another shell

 tail -f /var/log/auth.log | grep snoopy

Feb 23 07:52:38 debikos71 snoopy[12588]: [uid:0 sid:12485 tty: cwd:/root/src/snoopy-deb/lib filename:/bin/grep]: grep -i systemd
Feb 23 07:52:38 debikos71 snoopy[12589]: [uid:0 sid:12485 tty: cwd:/root/src/snoopy-deb/lib filename:/bin/grep]: grep ssh
Feb 23 07:52:38 debikos71 snoopy[12587]: [uid:0 sid:12485 tty:/dev/pts/1 cwd:/root/src/snoopy-deb/lib filename:/bin/cat]: cat /var/log/messages