I just have to do a post about a benefit of using log2timeline, because this is entirely too cool.
On a recent investigation, one system had a Seagate FreeAgent Go (USB HD) attached at some point, and this showed up in USB history (from Woanware USB Device Forensics and RegRipper). However, I could not associate it with the user profile through MountPoints2 (MP2) because the USBSTOR didn't contain the Parent Prefix ID Prefix (PPID) for this device. The PPID is used in MountedDevices to identify the drive letter association and GUID; the GUID is then used in MP2 to identify the user (without the GUID you get nothing). Basically all I had was the Vendor, Product, and Version from USBSTOR, along with the connection date from DeviceClasses. As a side note, Enum\USB also did not provide the Volume ID (VID) or Product ID (PID).
At first I wasn't sure why the parsed MP2 didn't show the user association, so I went through the manual process (as detailed by SANS, Harlan Carvey, etc). This revealed what I've noted above regarding the PPID. Even though I did not necessarily need to, I also went through the setupapi.log just to confirm install date, and work the steps, as it were.
To quickly find the device in setupapi, I searched the file for "FreeAgent" using Notepad++. By clicking the "Find All in Current Document" button (as opposed to "Find Next"), I saw that there were hits in very different locations within the file. Looking more closely at these entries, I discovered that there had in fact been two Seagate FreeAgent Go drives attached to the system, more than a month apart. Well, well, well. The serial number for the second one (as logged in setupapi, which also showed the VID and PID) did show up in RegRipper's output for USBSTOR and DeviceClasses, but again that wasn't everything.
I must note at this point that until I started going through manually I hadn't spent a lot of time on RR's output, having focused on Woanware with its nicely organized text file and spreadsheet for ease of use as I had a lot of systems and not a lot of time. I had confirmed the first drive's existence, but hadn't looked to see if there were more like it. Good reminder to cross-check results, even though it may take significantly more time, depending on the amount of relevant data...
Anyway, all that said, here's the cool part with l2t. I loaded up my timeline (created by l2t's CSV output module) in Excel and went to the date/time of the initial setupapi.log entry. Sure enough, there were the install entries, just like expected. And immediately following the installation activity came the MountPoints2 key entry parsed from System Restore Point RPxxx, showing the user association. This was true for both drives in question. In addition, these historic MP2 entries showed the GUID, to help round out the analysis. Aaah, the sweet smell of forensication in the morning... ;-).
Call me a fanboy if you must, but I do like log2timeline. Thanks, Kristinn!
LM
Thursday, April 28, 2011
Wednesday, April 20, 2011
Date-Range Specific Timelines
In the prior posts I built a timeline with the new mactime- and csv-format options in log2timeline, I used my bodyfile as input and exported to a nice csv that includes more l2t-related data (as opposed to standard mactime-generated csv). This is nice, but l2t generates one line per timestamp type and gives us a whole lot of rows. Turns out my main timeline was just too much data to be able to load in Excel or Access. Even with 64-bit Win7 and 16 GB of RAM on a Quad-Core processor. Apparently 1.2M rows is just insane. Meh.
I tried to get Kristinn to do things differently, but he just wouldn't see it "my" way. ;) However, in true form, he was extremely helpful. We walked through using grep against an l2t-generated csv to get out some specific sub-ranges that I needed to focus on, like:
This will split out all rows containing data originating (any timestamp type) from February through May of 2009.
Lo and behold! Kristinn also went and created a new module in the nightly build and asked me to help test it. It would automate the process, so you can feed in a date range (kind of like with mactime) and it will pull those dates out of an l2t-generated csv file. Way cool! And so the testing began.
Fortunately by that time I knew my test-set inside and out, so that was helpful in providing him statistics to help track down a few items. In the end, the process was ironed out and works very nicely. I'm not here to post all the specifics, as I know he plans to include it in the next release, and blog about it as well, so I'm leaving that to him. It's his baby, after all. I'm just trying to raise more awareness of this awesome utility, a cool developer, and get more traffic on his sites. Every time I've had a question or problem/concern, Kristinn has been extremely responsive and helpful. Twice now he's written new modules just because I asked if it could be done. Incredible!
I don't think he'll mind me saying that if you're using the nightly build, look for l2t_process; it even has its own man page.
Happy timelining!
LM
I tried to get Kristinn to do things differently, but he just wouldn't see it "my" way. ;) However, in true form, he was extremely helpful. We walked through using grep against an l2t-generated csv to get out some specific sub-ranges that I needed to focus on, like:
head -1 csv.001 > dates.001
grep “^0[2-5]\/[0-3][0-9]\/2009,” csv.001 >> dates.001 This will split out all rows containing data originating (any timestamp type) from February through May of 2009.
Lo and behold! Kristinn also went and created a new module in the nightly build and asked me to help test it. It would automate the process, so you can feed in a date range (kind of like with mactime) and it will pull those dates out of an l2t-generated csv file. Way cool! And so the testing began.
Fortunately by that time I knew my test-set inside and out, so that was helpful in providing him statistics to help track down a few items. In the end, the process was ironed out and works very nicely. I'm not here to post all the specifics, as I know he plans to include it in the next release, and blog about it as well, so I'm leaving that to him. It's his baby, after all. I'm just trying to raise more awareness of this awesome utility, a cool developer, and get more traffic on his sites. Every time I've had a question or problem/concern, Kristinn has been extremely responsive and helpful. Twice now he's written new modules just because I asked if it could be done. Incredible!
I don't think he'll mind me saying that if you're using the nightly build, look for l2t_process; it even has its own man page.
Happy timelining!
LM
Thursday, March 10, 2011
Timeline Registry Automation Script
I just couldn't keep from writing the script. Just couldn't. After going through the process yesterday, then posting about it, I just kept thinking about how I could streamline the process, so I took me a little bit of time and worked up the following. Ran through it a few times to make sure it worked and tweaked it a little.
Please keep in mind, I'm no scripting guru (aka, Hal Pomeranz et al) , so this may seem kludgy. But it does work. I could probably feed it a list of mount points and output files to fill in the variables and have it run through the whole of it, but that would probably take me more time to create and test (and fix) than it would for me to run this a handful of times.
So here it is, in all its (lack of) glory:
Here's hoping someone can use it.
Cheers!
LM
Please keep in mind, I'm no scripting guru (aka, Hal Pomeranz et al) , so this may seem kludgy. But it does work. I could probably feed it a list of mount points and output files to fill in the variables and have it run through the whole of it, but that would probably take me more time to create and test (and fix) than it would for me to run this a handful of times.
So here it is, in all its (lack of) glory:
_________________________________________________________________
#!/bin/sh
#
# Script to automate regripper in linux for timeline creation.
# This is designed to be run from your regripper directory.
# This version of rip.pl is brought over from the Windows download to run in Linux based on http://grey-corner.blogspot.com/2010/04/running-regripper-on-linux.html
#
# This will will automatically run through the 4 hives in a given mount point and write specified output file.
# By default, the 'all' module is run, rather than specific to hive type.
#
# $Src is the path (mount point) to be recursed for file in question
# $Dst is the path & file for regripper output (path must already exist)
# Order of operation should be ./rip.sh src dst
Src=$1
Dst=$2
#
#
# Check that the user provided all arguments required by this script.
if [ -z $1 ]; then
echo -e "USAGE: rip.sh SOURCEDIR OUTPUTFILE";
exit;
fi
if [ -z $2 ]; then
echo "USAGE: rip.sh sourcedir TARGETFILE";
exit;
fi
echo
echo
#
# Begin the job, updating the user along the way.
echo "Parsing user hive ... Please be patient."
echo
find $1 -iname ntuser.dat | while read d; do ./rip.pl -f all -r "$d" >> $2; done
echo
echo "Thank you for being patient."
echo
echo
echo "Parsing system hive ... This will only take a minute."
echo
find $1 -iname system | while read d; do ./rip.pl -f all -r "$d" >> $2; done
echo
echo "See, I told you it wouldn't take long."
echo
echo
echo "Parsing security hive ... Just a second, it's almost done."
echo
find $1 -iname sam | while read d; do ./rip.pl -f all -r "$d" >> $2; done
echo
echo "There! I can't believe you're so impatient."
echo
echo "Last one - the software hive ... Hold your horses, okay?"
echo
find $1 -iname software | while read d; do ./rip.pl -f all -r "$d" >> $2; done
echo
echo "Okay, we're done now. Stop complaining; I worked as fast as I could."
echo
echo
echo "If you want to run another system, please start over"
echo
echo "Thanks for playing; have a nice day."
echo
# end of script
#
_________________________________________________________________
Here's hoping someone can use it.
Cheers!
LM
Wednesday, March 9, 2011
Timelines with Registry Data
I've posted before about log2timeline, and now I'm going to add to that a bit.
This is about incorporating regripper output into the timeline. SANS teaches about this in FOR508, and I'm going to "streamline" the process a bit. Without SANS (where I learned of the possibilities) and Harlan Carvey (the source of regripper), and coffee this would not be possible.
So here's the situation: I'm working an investigation with several systems, and starting by building timelines. Standard process - fls, l2t, mactime. But before working up the final bodyfiles and running mactime, I'm going to pull in registry data with regripper.
I'm working in my linux system, which is Ubuntu based. Using Lupin's great post here I've incorporated the latest version of regripper into my system.
At first I was going to bash it together with variables, but in the end it seemed like there would be four or five, and I thought that was too much typing across several images. It seemed more efficient to type my command line once, then repeat and modify the couple things I would need.
So my evidence items are numbered sequentially, like XYZ-001, XYZ-002, and so on. Their respective file systems are mounted accordingly on /mnt/001, /mnt/002, ... The drive containing the images is a Truecrypt volume, so it's mounted on /media/truecrypt1. I'm storing my output files in a separate directory for each evidence item, named for the number and custodian, like /media/truecrypt1/001_jones, /media/truecrypt1/002_smith, etc. The bodyfiles I've already created are named like 001.fls, 001.l2t so I can easily tell which is which. My regripper output will continue in that vein with 001.reg.
Now regripper documentation and SANS teach to use it by pathing out to each hive, then redirecting output somewhere. Something like:
Something like that. But that's too much typing for me. And like I said, I thought about scripting it, but there's just too many variables for what needs to be done - source path, source file, module, destination path, destination file, ack pfft!
So to pull a page from my old ways with log2timeline (before timescanner), I did the following, from within my regripper directory:
Then all I have to do is replace "system" with "sam" "software" and "ntuser.dat" and that takes care of 001. For 002 I change the "1" to "2", change "jones" to "smith" and I'm good to go. Far quicker than typing all the paths each time, or plugging in several variables. If I wanted to get more creative I probably could script it to run through the various hives for each set of variables, and then just put in the set for the next custodian system. That might be okay, but I didn't feel like going that far today. Now that I think about it, I believe I will tomorrow, though.
This is no command line kung-fu, but I like it; it's better than a bunch of typing anyway. What it does is within the /mnt/001/ directory, it searches for files named "system" with no case-sensitivity. STDOUT is the normal output; this is piped to a while loop that runs rip.pl against the "system" file (-r). I had some errors trying to use the specific modules (-f) and being lazy (just like not wanting to type) I decided just to try "all." This worked just fine, no more errors. You'll see as it runs that regtime is called; I believe this is what creates the mactime-formatted output.
That takes care of my timeline pieces. Now to get a single bodyfile for each system... Within my output directory (/media/truecrypt1/001_jones):
I'm now reversing the order, to keep my pieces separated. It helps me track the flow/progress as well, which I'll show next.
I'm trying to identify areas of activity (or inactivity) over several months so that I can focus in on details with a more limited timeframe. To do this I'll be running mactime and building a daily index. For 001, this looks like:
This is just wrong. Shame on me. Here's how it needs to be:
This gives me a CSV file (-d) that gives a nice overview of total activity each day during my ~4 month period. Then hopefully I'll be able to focus in on a few specific days that look interesting (maybe a lot of activity, maybe very little). All these options are in the help info or man pages, but in brief -d gives a CSV output, -z allows you to set the timezone, -m and -y set the date format, -b specifies a bodyfile, then comes the date range I'm interested in, and last -i specifies to generate an index; it's either "day" or "month", and must be redirected to the output file. Again, all I have to do is change "1" to "2", "jones" to "smith" and move on to the next.
Now my output directory has the following files:
001.fls, 001.l2t, 001.reg, body.001, index.001
This makes it easier for me to keep track of the different sets of data, and their formats. The pieces that go into my main bodyfile start numerically, and the combined datafiles end numerically. I know everyone has a different way; just wanted to share my logic/excuse.
I hope that can help someone. I should (hopefully) have some sort of shell script together tomorrow that would loop through the different hives for each custodian. There would still be several parameters to input initially, but it might work out to be faster going through four hives than up arrow, back arrow, replace... Once I have it I'll post that as well.
LM
Note: Edited to correct mactime syntax to create index files.
This is about incorporating regripper output into the timeline. SANS teaches about this in FOR508, and I'm going to "streamline" the process a bit. Without SANS (where I learned of the possibilities) and Harlan Carvey (the source of regripper), and coffee this would not be possible.
So here's the situation: I'm working an investigation with several systems, and starting by building timelines. Standard process - fls, l2t, mactime. But before working up the final bodyfiles and running mactime, I'm going to pull in registry data with regripper.
I'm working in my linux system, which is Ubuntu based. Using Lupin's great post here I've incorporated the latest version of regripper into my system.
At first I was going to bash it together with variables, but in the end it seemed like there would be four or five, and I thought that was too much typing across several images. It seemed more efficient to type my command line once, then repeat and modify the couple things I would need.
So my evidence items are numbered sequentially, like XYZ-001, XYZ-002, and so on. Their respective file systems are mounted accordingly on /mnt/001, /mnt/002, ... The drive containing the images is a Truecrypt volume, so it's mounted on /media/truecrypt1. I'm storing my output files in a separate directory for each evidence item, named for the number and custodian, like /media/truecrypt1/001_jones, /media/truecrypt1/002_smith, etc. The bodyfiles I've already created are named like 001.fls, 001.l2t so I can easily tell which is which. My regripper output will continue in that vein with 001.reg.
Now regripper documentation and SANS teach to use it by pathing out to each hive, then redirecting output somewhere. Something like:
# rip.pl -r /media/sda2/Windows/System32/config/SAM -f sam >> /media/sdb1/evidence/timelines/jones_regripper.txtSomething like that. But that's too much typing for me. And like I said, I thought about scripting it, but there's just too many variables for what needs to be done - source path, source file, module, destination path, destination file, ack pfft!
So to pull a page from my old ways with log2timeline (before timescanner), I did the following, from within my regripper directory:
# find /mnt/001/ -iname system | while read d; do ./rip.pl -f all -r "$d" >> /media/truecrypt1/001_jones/001.reg; doneThen all I have to do is replace "system" with "sam" "software" and "ntuser.dat" and that takes care of 001. For 002 I change the "1" to "2", change "jones" to "smith" and I'm good to go. Far quicker than typing all the paths each time, or plugging in several variables. If I wanted to get more creative I probably could script it to run through the various hives for each set of variables, and then just put in the set for the next custodian system. That might be okay, but I didn't feel like going that far today. Now that I think about it, I believe I will tomorrow, though.
This is no command line kung-fu, but I like it; it's better than a bunch of typing anyway. What it does is within the /mnt/001/ directory, it searches for files named "system" with no case-sensitivity. STDOUT is the normal output; this is piped to a while loop that runs rip.pl against the "system" file (-r). I had some errors trying to use the specific modules (-f) and being lazy (just like not wanting to type) I decided just to try "all." This worked just fine, no more errors. You'll see as it runs that regtime is called; I believe this is what creates the mactime-formatted output.
That takes care of my timeline pieces. Now to get a single bodyfile for each system... Within my output directory (/media/truecrypt1/001_jones):
# cat 001.* >> body.001I'm now reversing the order, to keep my pieces separated. It helps me track the flow/progress as well, which I'll show next.
I'm trying to identify areas of activity (or inactivity) over several months so that I can focus in on details with a more limited timeframe. To do this I'll be running mactime and building a daily index. For 001, this looks like:
# mactime -d -z CST6CDT -m -y -b /media/truecrypt1/001_jones/body.001 2010-01-30..2010-05-24 -i day >> /media/truecrypt1/001_jones/index.001This is just wrong. Shame on me. Here's how it needs to be:
# mactime -b /media/truecrypt1/001_jones/body.001 -i day /media/truecrypt1/001_jones/index.001 -d -m -y -z CST6CDT 2010-01-30..2010-05-24This gives me a CSV file (-d) that gives a nice overview of total activity each day during my ~4 month period. Then hopefully I'll be able to focus in on a few specific days that look interesting (maybe a lot of activity, maybe very little). All these options are in the help info or man pages, but in brief -d gives a CSV output, -z allows you to set the timezone, -m and -y set the date format, -b specifies a bodyfile, then comes the date range I'm interested in, and last -i specifies to generate an index; it's either "day" or "month", and must be redirected to the output file. Again, all I have to do is change "1" to "2", "jones" to "smith" and move on to the next.
Now my output directory has the following files:
001.fls, 001.l2t, 001.reg, body.001, index.001
This makes it easier for me to keep track of the different sets of data, and their formats. The pieces that go into my main bodyfile start numerically, and the combined datafiles end numerically. I know everyone has a different way; just wanted to share my logic/excuse.
I hope that can help someone. I should (hopefully) have some sort of shell script together tomorrow that would loop through the different hives for each custodian. There would still be several parameters to input initially, but it might work out to be faster going through four hives than up arrow, back arrow, replace... Once I have it I'll post that as well.
LM
Note: Edited to correct mactime syntax to create index files.
Tuesday, March 8, 2011
Need More Cowbell (or, links to SANS)
So Rob Lee recently posted to the DFIR list that SANS is ranking rather low on Google, as they're not stooping to "shadier" tactics such as setting up false sites to point to theirs, and other similar techniques. I say shadier since doing those kinds of SEO type things (aside from your site's internal metadata, search engine indexing, and so on) drives up your presence on the web - but at others' expense, and typically seems unrelated to the actual site. That may not make a lot of sense, but that's okay.
The point of this post is to say that I've done my part. I already follow the SANS Forensics Blog, but now I've added that and their main forensics page under Favorite Sites. Just doing my forensivic duty (forensivic=forensic civic).
On a side note, Rob suggested searching for "computer forensics" from Google to see the results. I did so, and was pleased to see Lance Mueller's site in the first page of results. Obviously he gets a lot of traffic, and well worth it!
LM
The point of this post is to say that I've done my part. I already follow the SANS Forensics Blog, but now I've added that and their main forensics page under Favorite Sites. Just doing my forensivic duty (forensivic=forensic civic).
On a side note, Rob suggested searching for "computer forensics" from Google to see the results. I did so, and was pleased to see Lance Mueller's site in the first page of results. Obviously he gets a lot of traffic, and well worth it!
LM
Thursday, March 3, 2011
The Whole HBGary v Anonymous Scenario
So it's a bit old news now, but it just doesn't seem to quit, and it's all rather interesting. And that is, of course, HBGary Federal (and HBGary's Greg Hoglund) being thoroughly smacked around by hacker group Anonymous. Ars Technica has some very in-depth coverage of the whole situation, with looks at what led up to it as well the aftermath.
Obviously there's a whole slew of questions about how a security company could have such seemingly glaring lapses in best practices, all the way around the board. But then too, it's easy to play armchair quarterback in hindsight. While I scratch my head about it, that's not why I'm posting.
From what Ars Technica has posted, there would also seem to be a lot of questions about ethical/moral considerations for other HBGary Federal activities. These things, when seen from the outside (again) and not knowing the whole facts/truth, would seem to be likely to lead to investigations into HBGary Federal by law enforcement as well as into Anonymous' activities. However, that's also not the reason for my post.
I'm posting because of the way the attacks got pulled off. Nothing fancy, cutting edge, or unique. Just a good old-fashioned SQL injection made possible by a lack of whitelisting. Password hashes extracted, cracked, and found to have multiple uses. Greg Hoglund's email being compromised and used to gain remote root access to rootkit.com through social engineering. Classic stuff, and (seemingly) fairly well executed - at least based on the results. Ars Technica published the email exchange between an Anonymous member posing as Hoglund, and Jussi Jaakonaho, wherein Jaakonaho was played in order to give Anonymous root access over ssh to take over the rootkit.com server.
To me, that last social engineering bit is the "sweet" piece. No, I'm not supporting Anonymous' actions, I'm just looking in through a window and thinking that from a technical standpoint, they did a good job. Once they got a foot in the door, they quickly went through a series of steps gaining more and more control over their target environment. Looking back at it, from the outside, it would appear that there were several opportunities for Jaakonaho to get suspicious and try to confirm through some other channel, but he didn't. All the other aspects - gaining access to the CMS, email, data storage, defacing websites, and so on, are all simply technical skills if you will.
The social engineering bit, though, stands out (to me). It was (virtually) face to face. They had to pretend to be Hoglund and communicate with someone who knew him, then try to get that person to do something that would seem to be against the very nature of a security-focused person. I mean, getting Jaakonaho to take down the firewall to open up ports for ssh access, reset passwords, hand out user names and public IP address - wow! To me, even trying to pull that off takes some serious guts. But the fact is, it worked, and quite nicely.
I don't condone what they did at all, but I guess I would have to say that I admire - at least that one piece, at least to an extent - *how* they did it.
In case anyone hasn't read about it and wants to, here are some of the Ars Technica links:
The Inside Story
The Aftermath
The Meet
The Email Revelation
One last little thought on the matter. A part of me can't help but wonder, is it possible that Aaron Barr/HBGary (Federal or otherwise) could have faked out Anonymous, and gave them a carefully orchestrated scenario? You know, a really seriously elaborate honeypot? Surely not. But I do wonder...
LM
Obviously there's a whole slew of questions about how a security company could have such seemingly glaring lapses in best practices, all the way around the board. But then too, it's easy to play armchair quarterback in hindsight. While I scratch my head about it, that's not why I'm posting.
From what Ars Technica has posted, there would also seem to be a lot of questions about ethical/moral considerations for other HBGary Federal activities. These things, when seen from the outside (again) and not knowing the whole facts/truth, would seem to be likely to lead to investigations into HBGary Federal by law enforcement as well as into Anonymous' activities. However, that's also not the reason for my post.
I'm posting because of the way the attacks got pulled off. Nothing fancy, cutting edge, or unique. Just a good old-fashioned SQL injection made possible by a lack of whitelisting. Password hashes extracted, cracked, and found to have multiple uses. Greg Hoglund's email being compromised and used to gain remote root access to rootkit.com through social engineering. Classic stuff, and (seemingly) fairly well executed - at least based on the results. Ars Technica published the email exchange between an Anonymous member posing as Hoglund, and Jussi Jaakonaho, wherein Jaakonaho was played in order to give Anonymous root access over ssh to take over the rootkit.com server.
To me, that last social engineering bit is the "sweet" piece. No, I'm not supporting Anonymous' actions, I'm just looking in through a window and thinking that from a technical standpoint, they did a good job. Once they got a foot in the door, they quickly went through a series of steps gaining more and more control over their target environment. Looking back at it, from the outside, it would appear that there were several opportunities for Jaakonaho to get suspicious and try to confirm through some other channel, but he didn't. All the other aspects - gaining access to the CMS, email, data storage, defacing websites, and so on, are all simply technical skills if you will.
The social engineering bit, though, stands out (to me). It was (virtually) face to face. They had to pretend to be Hoglund and communicate with someone who knew him, then try to get that person to do something that would seem to be against the very nature of a security-focused person. I mean, getting Jaakonaho to take down the firewall to open up ports for ssh access, reset passwords, hand out user names and public IP address - wow! To me, even trying to pull that off takes some serious guts. But the fact is, it worked, and quite nicely.
I don't condone what they did at all, but I guess I would have to say that I admire - at least that one piece, at least to an extent - *how* they did it.
In case anyone hasn't read about it and wants to, here are some of the Ars Technica links:
The Inside Story
The Aftermath
The Meet
The Email Revelation
One last little thought on the matter. A part of me can't help but wonder, is it possible that Aaron Barr/HBGary (Federal or otherwise) could have faked out Anonymous, and gave them a carefully orchestrated scenario? You know, a really seriously elaborate honeypot? Surely not. But I do wonder...
LM
Thursday, February 24, 2011
Eric Huber (A Fistful of Dongles) Interviews Hal Pomeranz
Eric Huber recently posted an interview with Hal Pomeranz on his blog, here. I'm not typically a person who reads interviews, but I had Hal as one of my SANS instructors. I think Hal's one of the many sharp forensic folks out there, and I've certainly learned a lot from him. So anyway, I was quite a bit more interested in “hearing” what he had to say. And because of this posting of Eric's, I've actually started reading more interviews. So hey, a positive impact!
I thought the interview was very good (thanks, Eric!) and there were a lot of good take-aways in there for me, and the company where I work (a boutique firm where we work primarily for lawfirms on behalf of corporations). If it was that good for me, then surely it's good for others, too. So I thought I'd share my – well, for lack of a better word – thoughts on what was said. I've been working on this post ever since the interview first came out (I know, it's taken a while – I'd like to blame my work schedule, if I may). I first wrote a version of this for my company (in a spreadsheet), as I thought we could (and should) take some of these things to heart. Obviously, I've made some changes to post to my own (infrequently updated) blog.
Eric's questions weren't numbered, but I've taken the liberty of referring to them as if they were, for ease of making sure I've covered my points, and so that it's easier to track back to the original.
1. Not much here. Eric asked about Hal's journey into forensics, and Hal mentioned being approached by Rob Lee to be part of Mandiant's Surge Staff. I just think it would be cool – an honor to work with Mandiant. Perhaps some day; a guy's gotta have dreams, after all … ;)
2. This question was about Hal's involvement with SANS' Forensics program, and Hal talked about how challenging it is to prepare to teach the material. I've seen the same thing in the military and business - when you have to prepare to teach a subject, you learn more about it than you probably ever would by simply taking the class yourself, or doing the work. Teaching is hard work, but obviously the rewards are extremely beneficial in this respect. In addition, getting outside of your environment and teaching (not just outside your comfort zone) helps establish your presence in the community at large (both personally and professionally).
Teaching others isn't just about personal (or even professional) accolades. It is an opportunity to learn even more about the craft/science we use, and giving back to the community. Forensics has developed as a community-driven discipline, because there aren't rulebooks, documentation, etc - we are the ones paving the way. As we benefit from the community, we really have a responsibility to give back, or "pay it forward" as Hal mentions. It's also a way we make connections in the industry which could lead to business associations as well. Later on, Hal emphasizes the importance of giving back to the community - doing blogs, whitepapers, articles, etc. I think that plays a part here as well. You don't have to be a Harlan Carvey, Didier Stevens, or Hal Pomeranz to contribute. Simple blog posts about the things you find can be useful, as you may have come across something somebody else hasn't seen before, and you may be the only person (or one of a very few) reporting on it. There's at least one blog I know of where the guy is new to forensics, but he's doing a lot in Linux and posting findings on his blog.
3. Eric asked about skills Hal has developed over the years that help with forensics, and Hal talked about tricks and other things he learned from his years as a Unix Sys Admin, scripting, and problem solving. This whole section speaks to what I think is really part of what goes into our team (where I work). We have varied backgrounds, different business paths, and so on - these all help shape our skills, troubleshooting/problem-solving abilities, and so on. We don't just think from one perspective; even if a single individual did (and I don't think we do, given what I've seen), the "group mind" takes it much further as we collaborate and share based on our personal experiences. Together, we are greater than the sum of our parts, and I think this is one of the things that really sets our team apart and allows us to accomplish more than a lot of others in the field.
4. There are several things in this question about what it takes to be a good forensic examiner. Hal mentions root cause analysis, writing skills, and passion for the gig. Rob Lee has a great article out there discussing what it takes to get started in digital forensics (http://computer-forensics.sans.org/blog/2010/08/20/getting-started-digital-forensics-what-takes). IMHO, it's a very good writeup. One of the things I took away from it is that he looks for a drive (no, not a hard drive) in the person to accomplish new things. He gives the example of a person who decides to learn to play golf one summer and devotes time & effort to doing just that. He said he asks candidates what they've done along those lines and uses that as a barometer to gauge their forensic potential. (Section: "Do You Have A Desire To Be An Expert?") I don't think it's so much about "what you know" as what you want to know and are driven to learn. You can have a good IT person who knows a lot about computers but would not make a good forensicator, as they might not really want to learn more, or get "outside the box" for their current set of knowledge.
The whole writing thing, well this is a bit of a personal pet peeve. Well, part of it anyway - the coherent documentation bit. A lot of the forensic reports (and white papers) that I've reviewed are, in my opinion (humble or otherwise), drivel. I think people are in way too much of a hurry, don't take time to do error-checking, editing, grammar/syntax correction, formatting, or making sure they are communicating their point for the intended audience. There also seems to be a general lack of understanding regarding some basic grammar rules (run-on sentences, for instance, is a big offender). Then beyond that you also have to take the target audience into account, as Hal mentions. Unfortunately, I'm not sure that quality report writing is something that can be taught; it's like the "silver tongue" - either you have it or you don't. Anyone can improve their skill, but to what level? And some, I think, will not ever have it (something that needs to be watched for in the hiring process).
5. The question about how to get into digital forensics. Yeah. There's been a pretty good discussion on one of the LinkedIn groups about this, but Hal hits some different points, focusing primarily on networking. Not sure what a switch and a CAT6 cable will get me but... This section is where Hal mentions doing your own blog posts, articles, etc. He's approaching it from the standpoint of getting your name out there, looking for jobs, and so on. However, this works not only for the individual, but to help promote the company too. I know Digital Detective and Guardian Forensics, for example, both host their own blogs. I understand, though, that if you don't keep it active, you just look foolish as a company (does that count for individuals, too? Thinking about my own here...). He also emphasizes the importance of continuously learning. Again, there aren't any all-in-one rulebooks or documentation for our corner of the InfoSec industry. Operating systems change rapidly; these change available artifacts in ways that are not typically documented. With each change, we have to figure out what to do next. In addition to these "obvious" changes to the landscape, cell phones are now really miniature computers that also make phone calls - not to mention tablets brought back by the iPad. Their operating systems are completely new things, with new data structures, artifacts, and so on. Not only do we have to figure them out, we also have to figure out how to access the data (think Android, iOS, etc) such that we can forensically acquire it in a useful manner. If we don't keep up, we lose...
6. I don't have a lot of input here either, for the question about seeking advanced education or degrees in digital forensics. I'm not convinced that the current programs are actually preparing graduates for the real world, but then that's probably true for any degree plan, and I also have not by any means seen everything that's out there.
7. Question 7 focuses on the “state of the union” so to speak, regarding forensics. The first three things Hal brings up are our general youth, developing solutions to problems, and sharing information. This ties in with my point about us paving the way, and the need to continue learning. Microsoft and Apple, for example, are certainly not in the business of providing information to the forensics community about how their various operating systems work, what changes are being made that would impact our professional lives, and so on. For example, as browsers are designed more and more to "protect" users' privacy, the storing of browser artifacts is something that has changed fairly dramatically in recent months. When it comes to things like the iPhone, as people figure out how to hack it for various purposes, Apple tries to block it in a new release of the iOS. Can't fault Apple, as they're trying to protect their users and interests; however, as we are often looking to utilize the same exploit, this negatively impacts our ability - and then Jonathan Zdiarski has to come up with a new script or process... ;)
From there Hal goes into cross-platform tools. He and I had an email conversation about this very thing a little while back. He'd posted some questions about Mac forensics/tools on the GCFA list, and did not get a lot of responses. I let him know about the tools that we've used, and we had a little back-and-forth off-list. One of the things we touched on then was how it seems the majority of tools for Mac forensics only run natively in Mac OS. While this may be a natural starting point, it can be difficult for folks who don't have immediate access to these platforms, but have reason to need to do analysis.
The final area in this question relates to, as Hal puts it, “bridge-building.” I just think this is a very good point - educate the legal community on what we do, and how we can help them. If we also become more educated on what they need, we will in turn be better prepared to help them in a meaningful way. As Hal mentions later on, he sometimes finds stuff he thinks is cool or interesting, but it may have absolutely no value for the client. I think it's important for us to step out of our comfort zone (or simply take our blinders off) to get a better grasp of what our client really needs (as opposed to what we want to give them).
8. The question here is about difference between IT- and LE-based forensics, and I think Hal has some excellent points (but of course, right?). We tend to get wrapped up in our own "little" world and don't realize that our clients are juggling a lot of projects as well. They may not remember exactly what we told them before, and may have some difficulty grasping the points we're trying to make. A little consideration will probably go a long way to help create a stronger, more mutually beneficial relationship. We should have a "take-away" focus to help us learn how to better do our jobs (not just the way we've always done them).
The second half of this brings up rabbit trails, and maybe a little scope creep. This is one of the biggest issues I think we face, when it comes to actual investigations (by “we” I mean our company in specific, although I realize this may have broader applicability). We are bad about going off on our own tangents, "just because we want to know." Maybe this is useful from the standpoint of educating ourselves, and if we aren't billing the client, what's the harm, right? But if it detracts from us meeting goals/deadlines that the client has, then we're doing everyone a disservice. We have to stay on task, focused, and try to make sure that we are meeting the client's needs. Sometimes I think where this starts is we see something the client hasn't mentioned, or says they don't think they're interested in. We *know* that it is valuable, and that they *need* the info - so we go digging to get it, to have it ready once they "come to their senses." This is probably not the best approach, and we need to do a better job understanding the client's scenario, what their end-goal is, and so on, so we can make valid recommendations to help reach that goal. We also need to do a better job helping the client understand the value of what we are offering - our expertise and potential testimony on the stand. After all, we're a consulting firm; we should *consult*.
9. Aah, scoping, planning, and then communications with the client. Okay, here goes... This ties in with the comments above. Our approach to a situation, the deliverable we're providing, and so on, should really be tailored to the individual needs of the matter. If we have a "fit in the box" approach, we do everyone involved a disservice. From this standpoint, there are probably times (however much we might not want to do so for whatever reasons) that we need to customize (or completely change) our User File approach (we have a proprietary process that starts with deNISTing and goes much much further; this is used in pre-culling for eDiscovery projects). One potential example that comes to mind is if a matter only stipulates that email, or MS Office documents, or PDF files are required, it might be reasonable to only produce those data types and save the client time and cost. Obviously, as Hal points out, communication with the client is key.
This is also true for investigations – while it's good to have an idea of a “standard” approach to a different type of investigation (misuse of resources, theft of IP, spoliation, etc), we need to make sure we understand what the client needs to accomplish, so we can tailor our approach to their needs. This way they get the benefit of the best possible results, and we don't waste time doing things that aren't going to help reach the goal.
So then we come to the communications part of Hal's response. Let's rub some more salt on that wound, why don't we (again, where I work we seem to struggle with this; some analysts don't seem to want to have a lot of interaction with the client)? We as analysts need to be better about direct communication with the client. There's a time to involve the account/project manager, but no one is better prepared to communicate technical details of our investigation than we are. And it's not just the aspect of making sure the client knows how we're progressing - it's the personal touch that makes a difference. The analyst, having ownership of the investigation, is able to make a personal connection with the client, and the client is going to remember them down the road (as Hal notes). Case in point, I have one attorney that sends me Christmas cards every year - and calls our company for more work - because of the relationship that developed while working on a case with him. And yes, I'm proud of that.
10. Wow, there's a lot here in this last section, and it is stuff that can probably get a lot of folks all fired up. I remember a while back we were asked to work defense on a criminal case. Regardless whether we wanted to turn it down or not, we were "required" to do so as a condition of our director's membership in the HTCIA - no criminal defense is allowed. I think Hal has a valid point in that everyone in this country is entitled to a defense, and it doesn't really seem "fair" that they be prevented from doing so by not being able to access the same resources (ie, forensicators) that are being brought against them. Obviously that gets into the political realm, but aside from that I think the most salient point he has is about scientific fact-finding being the foremost goal, not personal or professional agenda. We've certainly been involved in cases where the other side brought in their own experts, and a most contentious fight ensued as they weren't willing to admit that we were doing what needed to be done. These aren't criminal matters (since we don't typically work in that space), but the point is still the same - shouldn't we be working toward a common goal of fact-finding, rather than fighting about semantics (or other irrelevant issues)?
I guess on the other side of the issue, we try our best to do a top-shelf job, being professional and just looking for - and presenting - the facts. However, I've seen a number of firms that appear to be hired guns without ethics or morals, who are willing to say whatever their clients request, for the sake of a buck. To me, that's just sad.
Eric and Hal, thanks for a great interview and blog post. I enjoyed it. I hope others have as well. Just to give you a quick plug, yours is one of the (many) blogs I follow (and Command Line Kung Fu, of course). (Google Reader on my phone is a tremendous help during dead time while imaging at a client site!) I look forward to seeing new posts.
LM
I thought the interview was very good (thanks, Eric!) and there were a lot of good take-aways in there for me, and the company where I work (a boutique firm where we work primarily for lawfirms on behalf of corporations). If it was that good for me, then surely it's good for others, too. So I thought I'd share my – well, for lack of a better word – thoughts on what was said. I've been working on this post ever since the interview first came out (I know, it's taken a while – I'd like to blame my work schedule, if I may). I first wrote a version of this for my company (in a spreadsheet), as I thought we could (and should) take some of these things to heart. Obviously, I've made some changes to post to my own (infrequently updated) blog.
Eric's questions weren't numbered, but I've taken the liberty of referring to them as if they were, for ease of making sure I've covered my points, and so that it's easier to track back to the original.
1. Not much here. Eric asked about Hal's journey into forensics, and Hal mentioned being approached by Rob Lee to be part of Mandiant's Surge Staff. I just think it would be cool – an honor to work with Mandiant. Perhaps some day; a guy's gotta have dreams, after all … ;)
2. This question was about Hal's involvement with SANS' Forensics program, and Hal talked about how challenging it is to prepare to teach the material. I've seen the same thing in the military and business - when you have to prepare to teach a subject, you learn more about it than you probably ever would by simply taking the class yourself, or doing the work. Teaching is hard work, but obviously the rewards are extremely beneficial in this respect. In addition, getting outside of your environment and teaching (not just outside your comfort zone) helps establish your presence in the community at large (both personally and professionally).
Teaching others isn't just about personal (or even professional) accolades. It is an opportunity to learn even more about the craft/science we use, and giving back to the community. Forensics has developed as a community-driven discipline, because there aren't rulebooks, documentation, etc - we are the ones paving the way. As we benefit from the community, we really have a responsibility to give back, or "pay it forward" as Hal mentions. It's also a way we make connections in the industry which could lead to business associations as well. Later on, Hal emphasizes the importance of giving back to the community - doing blogs, whitepapers, articles, etc. I think that plays a part here as well. You don't have to be a Harlan Carvey, Didier Stevens, or Hal Pomeranz to contribute. Simple blog posts about the things you find can be useful, as you may have come across something somebody else hasn't seen before, and you may be the only person (or one of a very few) reporting on it. There's at least one blog I know of where the guy is new to forensics, but he's doing a lot in Linux and posting findings on his blog.
3. Eric asked about skills Hal has developed over the years that help with forensics, and Hal talked about tricks and other things he learned from his years as a Unix Sys Admin, scripting, and problem solving. This whole section speaks to what I think is really part of what goes into our team (where I work). We have varied backgrounds, different business paths, and so on - these all help shape our skills, troubleshooting/problem-solving abilities, and so on. We don't just think from one perspective; even if a single individual did (and I don't think we do, given what I've seen), the "group mind" takes it much further as we collaborate and share based on our personal experiences. Together, we are greater than the sum of our parts, and I think this is one of the things that really sets our team apart and allows us to accomplish more than a lot of others in the field.
4. There are several things in this question about what it takes to be a good forensic examiner. Hal mentions root cause analysis, writing skills, and passion for the gig. Rob Lee has a great article out there discussing what it takes to get started in digital forensics (http://computer-forensics.sans.org/blog/2010/08/20/getting-started-digital-forensics-what-takes). IMHO, it's a very good writeup. One of the things I took away from it is that he looks for a drive (no, not a hard drive) in the person to accomplish new things. He gives the example of a person who decides to learn to play golf one summer and devotes time & effort to doing just that. He said he asks candidates what they've done along those lines and uses that as a barometer to gauge their forensic potential. (Section: "Do You Have A Desire To Be An Expert?") I don't think it's so much about "what you know" as what you want to know and are driven to learn. You can have a good IT person who knows a lot about computers but would not make a good forensicator, as they might not really want to learn more, or get "outside the box" for their current set of knowledge.
The whole writing thing, well this is a bit of a personal pet peeve. Well, part of it anyway - the coherent documentation bit. A lot of the forensic reports (and white papers) that I've reviewed are, in my opinion (humble or otherwise), drivel. I think people are in way too much of a hurry, don't take time to do error-checking, editing, grammar/syntax correction, formatting, or making sure they are communicating their point for the intended audience. There also seems to be a general lack of understanding regarding some basic grammar rules (run-on sentences, for instance, is a big offender). Then beyond that you also have to take the target audience into account, as Hal mentions. Unfortunately, I'm not sure that quality report writing is something that can be taught; it's like the "silver tongue" - either you have it or you don't. Anyone can improve their skill, but to what level? And some, I think, will not ever have it (something that needs to be watched for in the hiring process).
5. The question about how to get into digital forensics. Yeah. There's been a pretty good discussion on one of the LinkedIn groups about this, but Hal hits some different points, focusing primarily on networking. Not sure what a switch and a CAT6 cable will get me but... This section is where Hal mentions doing your own blog posts, articles, etc. He's approaching it from the standpoint of getting your name out there, looking for jobs, and so on. However, this works not only for the individual, but to help promote the company too. I know Digital Detective and Guardian Forensics, for example, both host their own blogs. I understand, though, that if you don't keep it active, you just look foolish as a company (does that count for individuals, too? Thinking about my own here...). He also emphasizes the importance of continuously learning. Again, there aren't any all-in-one rulebooks or documentation for our corner of the InfoSec industry. Operating systems change rapidly; these change available artifacts in ways that are not typically documented. With each change, we have to figure out what to do next. In addition to these "obvious" changes to the landscape, cell phones are now really miniature computers that also make phone calls - not to mention tablets brought back by the iPad. Their operating systems are completely new things, with new data structures, artifacts, and so on. Not only do we have to figure them out, we also have to figure out how to access the data (think Android, iOS, etc) such that we can forensically acquire it in a useful manner. If we don't keep up, we lose...
6. I don't have a lot of input here either, for the question about seeking advanced education or degrees in digital forensics. I'm not convinced that the current programs are actually preparing graduates for the real world, but then that's probably true for any degree plan, and I also have not by any means seen everything that's out there.
7. Question 7 focuses on the “state of the union” so to speak, regarding forensics. The first three things Hal brings up are our general youth, developing solutions to problems, and sharing information. This ties in with my point about us paving the way, and the need to continue learning. Microsoft and Apple, for example, are certainly not in the business of providing information to the forensics community about how their various operating systems work, what changes are being made that would impact our professional lives, and so on. For example, as browsers are designed more and more to "protect" users' privacy, the storing of browser artifacts is something that has changed fairly dramatically in recent months. When it comes to things like the iPhone, as people figure out how to hack it for various purposes, Apple tries to block it in a new release of the iOS. Can't fault Apple, as they're trying to protect their users and interests; however, as we are often looking to utilize the same exploit, this negatively impacts our ability - and then Jonathan Zdiarski has to come up with a new script or process... ;)
From there Hal goes into cross-platform tools. He and I had an email conversation about this very thing a little while back. He'd posted some questions about Mac forensics/tools on the GCFA list, and did not get a lot of responses. I let him know about the tools that we've used, and we had a little back-and-forth off-list. One of the things we touched on then was how it seems the majority of tools for Mac forensics only run natively in Mac OS. While this may be a natural starting point, it can be difficult for folks who don't have immediate access to these platforms, but have reason to need to do analysis.
The final area in this question relates to, as Hal puts it, “bridge-building.” I just think this is a very good point - educate the legal community on what we do, and how we can help them. If we also become more educated on what they need, we will in turn be better prepared to help them in a meaningful way. As Hal mentions later on, he sometimes finds stuff he thinks is cool or interesting, but it may have absolutely no value for the client. I think it's important for us to step out of our comfort zone (or simply take our blinders off) to get a better grasp of what our client really needs (as opposed to what we want to give them).
8. The question here is about difference between IT- and LE-based forensics, and I think Hal has some excellent points (but of course, right?). We tend to get wrapped up in our own "little" world and don't realize that our clients are juggling a lot of projects as well. They may not remember exactly what we told them before, and may have some difficulty grasping the points we're trying to make. A little consideration will probably go a long way to help create a stronger, more mutually beneficial relationship. We should have a "take-away" focus to help us learn how to better do our jobs (not just the way we've always done them).
The second half of this brings up rabbit trails, and maybe a little scope creep. This is one of the biggest issues I think we face, when it comes to actual investigations (by “we” I mean our company in specific, although I realize this may have broader applicability). We are bad about going off on our own tangents, "just because we want to know." Maybe this is useful from the standpoint of educating ourselves, and if we aren't billing the client, what's the harm, right? But if it detracts from us meeting goals/deadlines that the client has, then we're doing everyone a disservice. We have to stay on task, focused, and try to make sure that we are meeting the client's needs. Sometimes I think where this starts is we see something the client hasn't mentioned, or says they don't think they're interested in. We *know* that it is valuable, and that they *need* the info - so we go digging to get it, to have it ready once they "come to their senses." This is probably not the best approach, and we need to do a better job understanding the client's scenario, what their end-goal is, and so on, so we can make valid recommendations to help reach that goal. We also need to do a better job helping the client understand the value of what we are offering - our expertise and potential testimony on the stand. After all, we're a consulting firm; we should *consult*.
9. Aah, scoping, planning, and then communications with the client. Okay, here goes... This ties in with the comments above. Our approach to a situation, the deliverable we're providing, and so on, should really be tailored to the individual needs of the matter. If we have a "fit in the box" approach, we do everyone involved a disservice. From this standpoint, there are probably times (however much we might not want to do so for whatever reasons) that we need to customize (or completely change) our User File approach (we have a proprietary process that starts with deNISTing and goes much much further; this is used in pre-culling for eDiscovery projects). One potential example that comes to mind is if a matter only stipulates that email, or MS Office documents, or PDF files are required, it might be reasonable to only produce those data types and save the client time and cost. Obviously, as Hal points out, communication with the client is key.
This is also true for investigations – while it's good to have an idea of a “standard” approach to a different type of investigation (misuse of resources, theft of IP, spoliation, etc), we need to make sure we understand what the client needs to accomplish, so we can tailor our approach to their needs. This way they get the benefit of the best possible results, and we don't waste time doing things that aren't going to help reach the goal.
So then we come to the communications part of Hal's response. Let's rub some more salt on that wound, why don't we (again, where I work we seem to struggle with this; some analysts don't seem to want to have a lot of interaction with the client)? We as analysts need to be better about direct communication with the client. There's a time to involve the account/project manager, but no one is better prepared to communicate technical details of our investigation than we are. And it's not just the aspect of making sure the client knows how we're progressing - it's the personal touch that makes a difference. The analyst, having ownership of the investigation, is able to make a personal connection with the client, and the client is going to remember them down the road (as Hal notes). Case in point, I have one attorney that sends me Christmas cards every year - and calls our company for more work - because of the relationship that developed while working on a case with him. And yes, I'm proud of that.
10. Wow, there's a lot here in this last section, and it is stuff that can probably get a lot of folks all fired up. I remember a while back we were asked to work defense on a criminal case. Regardless whether we wanted to turn it down or not, we were "required" to do so as a condition of our director's membership in the HTCIA - no criminal defense is allowed. I think Hal has a valid point in that everyone in this country is entitled to a defense, and it doesn't really seem "fair" that they be prevented from doing so by not being able to access the same resources (ie, forensicators) that are being brought against them. Obviously that gets into the political realm, but aside from that I think the most salient point he has is about scientific fact-finding being the foremost goal, not personal or professional agenda. We've certainly been involved in cases where the other side brought in their own experts, and a most contentious fight ensued as they weren't willing to admit that we were doing what needed to be done. These aren't criminal matters (since we don't typically work in that space), but the point is still the same - shouldn't we be working toward a common goal of fact-finding, rather than fighting about semantics (or other irrelevant issues)?
I guess on the other side of the issue, we try our best to do a top-shelf job, being professional and just looking for - and presenting - the facts. However, I've seen a number of firms that appear to be hired guns without ethics or morals, who are willing to say whatever their clients request, for the sake of a buck. To me, that's just sad.
Eric and Hal, thanks for a great interview and blog post. I enjoyed it. I hope others have as well. Just to give you a quick plug, yours is one of the (many) blogs I follow (and Command Line Kung Fu, of course). (Google Reader on my phone is a tremendous help during dead time while imaging at a client site!) I look forward to seeing new posts.
LM
Subscribe to:
Posts (Atom)