Date-Range Specific Timelines

In the prior posts I built a timeline with the new mactime- and csv-format options in log2timeline, I used my bodyfile as input and exported to a nice csv that includes more l2t-related data (as opposed to standard mactime-generated csv). This is nice, but l2t generates one line per timestamp type and gives us a whole lot of rows. Turns out my main timeline was just too much data to be able to load in Excel or Access. Even with 64-bit Win7 and 16 GB of RAM on a Quad-Core processor. Apparently 1.2M rows is just insane. Meh.

I tried to get Kristinn to do things differently, but he just wouldn't see it "my" way. ;) However, in true form, he was extremely helpful. We walked through using grep against an l2t-generated csv to get out some specific sub-ranges that I needed to focus on, like:

head -1 csv.001 > dates.001
grep “^0[2-5]\/[0-3][0-9]\/2009,” csv.001 >> dates.001

This will split out all rows containing data originating (any timestamp type) from February through May of 2009.

Lo and behold! Kristinn also went and created a new module in the nightly build and asked me to help test it. It would automate the process, so you can feed in a date range (kind of like with mactime) and it will pull those dates out of an l2t-generated csv file. Way cool! And so the testing began.

Fortunately by that time I knew my test-set inside and out, so that was helpful in providing him statistics to help track down a few items. In the end, the process was ironed out and works very nicely. I'm not here to post all the specifics, as I know he plans to include it in the next release, and blog about it as well, so I'm leaving that to him. It's his baby, after all. I'm just trying to raise more awareness of this awesome utility, a cool developer, and get more traffic on his sites. Every time I've had a question or problem/concern, Kristinn has been extremely responsive and helpful. Twice now he's written new modules just because I asked if it could be done. Incredible!

I don't think he'll mind me saying that if you're using the nightly build, look for l2t_process; it even has its own man page.

Happy timelining!

LM

Timeline Registry Automation Script

I just couldn't keep from writing the script. Just couldn't. After going through the process yesterday, then posting about it, I just kept thinking about how I could streamline the process, so I took me a little bit of time and worked up the following. Ran through it a few times to make sure it worked and tweaked it a little.

Please keep in mind, I'm no scripting guru (aka, Hal Pomeranz et al) , so this may seem kludgy. But it does work. I could probably feed it a list of mount points and output files to fill in the variables and have it run through the whole of it, but that would probably take me more time to create and test (and fix) than it would for me to run this a handful of times.

So here it is, in all its (lack of) glory:



_________________________________________________________________

#!/bin/sh
#
# Script to automate regripper in linux for timeline creation.
# This is designed to be run from your regripper directory.
# This version of rip.pl is brought over from the Windows download to run in Linux based on http://grey-corner.blogspot.com/2010/04/running-regripper-on-linux.html
#
# This will will automatically run through the 4 hives in a given mount point and write specified output file.
# By default, the 'all' module is run, rather than specific to hive type.
#
# $Src is the path (mount point) to be recursed for file in question
# $Dst is the path & file for regripper output (path must already exist)
# Order of operation should be ./rip.sh src dst
Src=$1
Dst=$2
#
#
# Check that the user provided all arguments required by this script.
if [ -z $1 ]; then
echo -e "USAGE: rip.sh SOURCEDIR OUTPUTFILE";
exit;
fi
if [ -z $2 ]; then
echo "USAGE: rip.sh sourcedir TARGETFILE";
exit;
fi

echo

echo

#
# Begin the job, updating the user along the way.

echo "Parsing user hive ... Please be patient."

echo

find $1 -iname ntuser.dat | while read d; do ./rip.pl -f all -r "$d" >> $2; done

echo

echo "Thank you for being patient."

echo

echo

echo "Parsing system hive ... This will only take a minute."

echo

find $1 -iname system | while read d; do ./rip.pl -f all -r "$d" >> $2; done

echo

echo "See, I told you it wouldn't take long."

echo

echo

echo "Parsing security hive ... Just a second, it's almost done."

echo

find $1 -iname sam | while read d; do ./rip.pl -f all -r "$d" >> $2; done

echo

echo "There! I can't believe you're so impatient."

echo

echo "Last one - the software hive ... Hold your horses, okay?"

echo

find $1 -iname software | while read d; do ./rip.pl -f all -r "$d" >> $2; done

echo

echo "Okay, we're done now. Stop complaining; I worked as fast as I could."

echo
echo

echo "If you want to run another system, please start over"

echo

echo "Thanks for playing; have a nice day."
echo

# end of script
#

_________________________________________________________________


Here's hoping someone can use it.

Cheers!

LM

Timelines with Registry Data

I've posted before about log2timeline, and now I'm going to add to that a bit.

This is about incorporating regripper output into the timeline. SANS teaches about this in FOR508, and I'm going to "streamline" the process a bit. Without SANS (where I learned of the possibilities) and Harlan Carvey (the source of regripper), and coffee this would not be possible.

So here's the situation: I'm working an investigation with several systems, and starting by building timelines. Standard process - fls, l2t, mactime. But before working up the final bodyfiles and running mactime, I'm going to pull in registry data with regripper.

I'm working in my linux system, which is Ubuntu based. Using Lupin's great post here I've incorporated the latest version of regripper into my system.

At first I was going to bash it together with variables, but in the end it seemed like there would be four or five, and I thought that was too much typing across several images. It seemed more efficient to type my command line once, then repeat and modify the couple things I would need.

So my evidence items are numbered sequentially, like XYZ-001, XYZ-002, and so on. Their respective file systems are mounted accordingly on /mnt/001, /mnt/002, ... The drive containing the images is a Truecrypt volume, so it's mounted on /media/truecrypt1. I'm storing my output files in a separate directory for each evidence item, named for the number and custodian, like /media/truecrypt1/001_jones, /media/truecrypt1/002_smith, etc. The bodyfiles I've already created are named like 001.fls, 001.l2t so I can easily tell which is which. My regripper output will continue in that vein with 001.reg.

Now regripper documentation and SANS teach to use it by pathing out to each hive, then redirecting output somewhere. Something like:
# rip.pl -r /media/sda2/Windows/System32/config/SAM -f sam >> /media/sdb1/evidence/timelines/jones_regripper.txt
Something like that. But that's too much typing for me. And like I said, I thought about scripting it, but there's just too many variables for what needs to be done - source path, source file, module, destination path, destination file, ack pfft!

So to pull a page from my old ways with log2timeline (before timescanner), I did the following, from within my regripper directory:
# find /mnt/001/ -iname system | while read d; do ./rip.pl -f all -r "$d" >> /media/truecrypt1/001_jones/001.reg; done
Then all I have to do is replace "system" with "sam" "software" and "ntuser.dat" and that takes care of 001. For 002 I change the "1" to "2", change "jones" to "smith" and I'm good to go. Far quicker than typing all the paths each time, or plugging in several variables. If I wanted to get more creative I probably could script it to run through the various hives for each set of variables, and then just put in the set for the next custodian system. That might be okay, but I didn't feel like going that far today. Now that I think about it, I believe I will tomorrow, though.

This is no command line kung-fu, but I like it; it's better than a bunch of typing anyway. What it does is within the /mnt/001/ directory, it searches for files named "system" with no case-sensitivity. STDOUT is the normal output; this is piped to a while loop that runs rip.pl against the "system" file (-r). I had some errors trying to use the specific modules (-f) and being lazy (just like not wanting to type) I decided just to try "all." This worked just fine, no more errors. You'll see as it runs that regtime is called; I believe this is what creates the mactime-formatted output.

That takes care of my timeline pieces. Now to get a single bodyfile for each system... Within my output directory (/media/truecrypt1/001_jones):
# cat 001.* >> body.001
I'm now reversing the order, to keep my pieces separated. It helps me track the flow/progress as well, which I'll show next.

I'm trying to identify areas of activity (or inactivity) over several months so that I can focus in on details with a more limited timeframe. To do this I'll be running mactime and building a daily index. For 001, this looks like:

# mactime -d -z CST6CDT -m -y -b /media/truecrypt1/001_jones/body.001 2010-01-30..2010-05-24 -i day >> /media/truecrypt1/001_jones/index.001
This is just wrong. Shame on me. Here's how it needs to be:

# mactime -b /media/truecrypt1/001_jones/body.001 -i day /media/truecrypt1/001_jones/index.001 -d -m -y -z CST6CDT 2010-01-30..2010-05-24

This gives me a CSV file (-d) that gives a nice overview of total activity each day during my ~4 month period. Then hopefully I'll be able to focus in on a few specific days that look interesting (maybe a lot of activity, maybe very little). All these options are in the help info or man pages, but in brief -d gives a CSV output, -z allows you to set the timezone, -m and -y set the date format, -b specifies a bodyfile, then comes the date range I'm interested in, and last -i specifies to generate an index; it's either "day" or "month", and must be redirected to the output file. Again, all I have to do is change "1" to "2", "jones" to "smith" and move on to the next.

Now my output directory has the following files:
001.fls, 001.l2t, 001.reg, body.001, index.001
This makes it easier for me to keep track of the different sets of data, and their formats. The pieces that go into my main bodyfile start numerically, and the combined datafiles end numerically. I know everyone has a different way; just wanted to share my logic/excuse.

I hope that can help someone. I should (hopefully) have some sort of shell script together tomorrow that would loop through the different hives for each custodian. There would still be several parameters to input initially, but it might work out to be faster going through four hives than up arrow, back arrow, replace... Once I have it I'll post that as well.

LM

Note: Edited to correct mactime syntax to create index files.

Using Log2Timeline

Log2Timeline and forensic timeline creation


Creating timeline w/mmls, fls, log2timeline
This is really written with an image of a Windows system in mind. This was written as a guide for our lab (and me, to help remember), so keep in mind it's not necessarily intended to be a polished presentation. Some things were added/changed as my use of l2t progressed. The big difference between this and what has been published on the SANS blog and on Kristinn Gudjonsson's site is the use of 'find' and 'while' loops to recurse through directory structure instead of (for instance) going into each user profile for the ntuser.dat file.

These l2t cli bits of this are largely irrelevant now, with the introduction of the timescanner front-end (which does a great job of automating everything for you). However, there may (hopefully) be some other things of interest/use in different scenarios.

You do not have to extract the partition, but it's helpful to have a single image file for mounting purposes (I'm not sure about mounting a split image; at the least it will be easier to have a single/concatenated file)...

NOTE: Where something is enclosed such as [imagename] this includes full path to that file where needed; the "name" is a variable assigned by the analyst.

You need to gather partition info, which can be done with mmls or fdisk:

mmls [imagename] - this will show partition info (even from split image files), and you can identify whichever partition(s) you need (specifically you need the start point).
or
fdisk -ul [imagename]

To create the baseline bodyfile, use the following:
fls -m [original mountpoint, ie C:, /boot, etc] -r -f [file system type, type 'fls -f list' to find the values] -i raw -o [offset/start sector info from mmls] [path to imagefile] > [path to new bodyfile]
NOTE: The offset here is the straight info provided by mmls; it is not multiplied by 512 (sector info)

Variables here are:
C: - this is however you want it to show in the bodyfile for drive letter; keep in mind if you are working multiple partitions... (-m is what allows you to set this parameter)
ntfs - this will depend on the file system in question (-f denotes you will provide filesystem)
raw - this will depend on the image type (-i denotes you will provide image type)
offset info - as stated, this comes from mmls or fdisk (-o denotes you will provide offset)
recursive - this is denoted by (-r) which says you are recursing into all subdirectories

Bodyfiles are converted to timeline format w/mactime command (you don't do this now; it will happen at the end):
mactime -b [bodyfile] -d -m -y -z [timezone] 2000-01-31..2000-02-01 > [timelinefile]
Variables here are:
-m and -y: go together to establish date format - yyyy mm dd and day of week
-z: sets the timezone of the original system/image
-b: you will define the bodyfile to be used
-d: provides output in CSV format (this will add the timestamp to every line, even if it's the same second - you need this!)
dates: date range provided in yyyy-mm-dd..yyyy-mm-dd format - must use both, even for single day (in which case, the 2nd date MUST be one full day after) such as:2009-11-10..2009-11-11
NOTE: For original/main bodyfile, you won't use the date parameters; you want the entire system bodyfile intact (you will split out later if needed)
NOTE: There are some cases where you are having to process multiple bodyfiles into a single timelinefile, and want to use >> to append instead of overwrite. Be careful!

Creating log2timeline bodyfiles:
You can use the following to get lists of l2t variables/parameters:
log2timeline -f list

First you must mount the imageset loopback to be able to browse the filesystem for use w/l2t:
Multiply the Offset info (from mmls or fdisk) by the number of bytes per sector (also from mmls or fdisk), typically 512 (such as 63*512=32256)
mount -o ro,loop,offset=32256 -t auto [imagename] [mount point]
Variables here are:
offset: this will depend on the result of your partition math, as noted above

NOTE: I use mkdir to create my mountpoint, using things like "src" for source and "dst" for destination instead of relying on drive lettering (such as sdc1)

Now you can browse the mounted filesystem for your imageset, in order to create bodyfiles for l2t...

cd to any relevant profile, for ntuser.dat files
log2timeline -f userassist NTUser.dat > [UAbodyfile]
or from within Documents & Settings, do:
find . -iname ntuser.dat | while read d; do log2timeline -f userassist "$d" >> [UAbodyfile]; done ****This works better - more automated****

while within profile, run LNK files (if there are multiple users, do so within Documents & Settings):
find . -iname *.lnk | while read d; do log2timeline -f win_link "$d" >> [LNKbodyfile]; done

while within profile, run IE history (if there are multiple users, do so within Documents & Settings):
find . -iname index.dat | while read d; do log2timeline -f iehistory "$d" >> [IEbodyfile]; done

while within profile, run Firefox 3 history drill down into Application Data/Mozilla/Firefox/Profiles/xxxxxxxx.default and do:
log2timeline -f firefox3 places.sqlite > [FFbodyfile]
or from within Documents & Settings, do:
find . -iname places.sqlite | while read d; do log2timeline -f firefox3 "$d" >> [FFbodyfile]; done ****This works better - more automated****

cd to Recycler directory, and down into each S-1-5...
log2timeline -f recycler INFO2 > [RBbodyfile]
or from within RECYCLER directory, do:
find . -iname INFO2 | while read d; do log2timeline -f recycler "$d" >> [RBbodyfile]; done ****This works better - more automated****

cd to System Volume Information/_Restore/RP... for restore point info (you do not have to go into each subfolder)
log2timeline -f restore . >> [RPbodyfile]

cd to Windows/Prefetch directory for prefetch info
log2timeline -f prefetch . >> [PFbodyfile]

With all of your individual pieces, create a copy of your original system bodyfile, and add the l2t bodyfiles into it:
cp [bodyfile] [bodyfile2]
cat [XXbodyfile] >> [bodyfile2]
Repeat for each l2t bodyfile (denoted by the "XXbodyfile")

You now have a single bodyfile which contains all the l2t variables/optional bodyfile info. This is the point you will start splitting out timeline sections with date ranges:
mactime -b [bodyfile] -d -m -y -z [timezone] 2000-01-31..2000-02-01 > [timelinefile]

___________________

PS: I am very disappointed that Kristinn did not get the recognition he deserves through this year's Forensic4Cast awards. Log2Timeline is by far and away (in my book) the absolute best software contribution to digital forensics, and should have won that category hands down!