The Case for ... Investigating

Lee Whitfield posted on his corporate blog earlier today about reasons not to bring cyber investigations for litigation (specifically forensics and eDiscovery) in-house, in response to this article making a case for the opposite. I replied on twitter, and was promptly lambasted by Kyle Maxwell for not blogging about it instead. You know how it is, twitter just doesn't provide a good platform for detailed response, and Kyle seemed to feel that sending several tweets was inappropriate. If it went any further, I think he would've called me old and feable. Again. :( So here's my detailed post.

I think (hope) that my perspective is somewhat unique, as I have been on both sides of the fence - in consulting and corporate roles. I've spent a lot of time scoping these types of matters, talked with GCs, OCs, IT, InfoSec, and eDiscovery folks. Full disclosure, I worked for several years at the consulting firm where Lee now works (although I've never had the pleasure of working with him), and in my current corporate position I'm responsible for building out the in-house programs. Lee's company focuses on reducing datasets so that eDiscovery costs are lower. That service is nowhere near the eDisco costs, but it ain't free either. ;) I was brought on where I am now specifically because they wanted the internal capabilities, having dealt with the difficulties of not having that kind of expertise in-house, and the tremendous cost of paying outside consultants and vendors.

In the original article, which was based primarily on an interview with Greg Thompson of Scotiabank, the essence of Thompson's argument is that it's totally worth it. He stated that they can easily spent $2000/day with an external vendor, compared to internal costs of $800/day. That's a pretty obvious ROI. My guess is that they're talking about an all-in-one vendor that collects, processes, and produces the data, with reductions coming only from deduping and deNISTing, and that probably occurs only after loading in the eDisco software.

Lee has a three-point rebuttal that centers on: Cost, Impartiality, and Skill set (not Skil saw, that's very different). Lee's company is not an eDisco processor, they're a forensic shop whose bread and butter is large-scale collection and data reduction for litigation. No, I'm not selling them, but as I said, I used to work there, so I'm very familiar. Because of the culling process they employ, data sets can be significantly reduced compared to what was collected, with associated lowering of eDiscovery processing costs.

My basic response to Lee was, "It depends." Trying to expound on that in twitter is pretty much fail, so I'll attempt to do so here. Basically, you've got a "party line" on either side. No, I'm not talking about the old phone system where you could pick up your home receiver and hear your neighbors' conversation. I'm talking about the territorial/turf war approach. In general terms. The consultants say if you're not out-sourcing, you're not doing it right and risk sanctions. The corporate folks say that they don't need somebody coming in "administering" their network and charging money for something they can do just fine. From my experience on both sides of the fence, here's how I think it breaks down.

Cost:
In a nutshell, Lee points to the salary of the kind of experienced expertise you'll need, software/hardware, training, and certifications. That's true, it's not cheap to get that kind of personnel. There's a couple things with this though, that I think bear more discussion. As with any such position (even in consulting), it's probably not a dedicated role, so the actual percentage of salary that applies to the forensics/eDisco work is not anywhere close to the six figures Lee mentions. Anyone in IT (much less InfoSec) is going to require ongoing training and certifications, and any employer that places value on professional development will support that anyway. So that only leaves us with hardware/software, which up front may be a sizable layout in cost, but will pay for itself very rapidly. How so? Well, a single case with 30 or more custodians could quickly cost over $100K. If you have one or more of those per year, your internal programs are covered. ROI's easy there.

Impartiality:
This may - in my opinion - be the best argument, up to a point. The corporation is paying the internal resource's paycheck, so those individuals have to support the corporate position, right? Not a bad assumption, but the same can apply to a consultant firm - they're paid by some organization to act in support of that org; if they don't give "good" results, they're out, right? So that knife cuts both ways, I think. But to the original point, I think it comes down to ethics of the investigator, just as with any case; we have an ethical, professional, and moral responsibility to do what is right, no matter what. Since the core of our work is based on facts in evidence, this shouldn't be an issue (at least not theoretically, but again, that cuts both ways). I think in most cases, an internal investigation is acceptable; there may be times that is different, and those should be addressed accordingly. The company - and its investigators - need to be able to determine when it may not be appropriate for the investigation to be handled internally. I know Kyle has mentioned having to deal with that where he works.

Skill set:
I'm a little confused on this one, to be honest. Lee says that most in-house investigators come from security or investigative backgrounds, discusses that network forensics has little to do with host forensics or eDiscovery, then goes on to say that while having "IT" staff involved, they shouldn't necessarily collect data themselves, as that could stomp on its evidential value. Okay, that's a long sentence, and a paraphrase of several combined. My confusion comes in from his starting out talking about computer security, network security, investigation background, and network forensics, then pointing out that IT staff aren't trained to know about file system changes, timestamps, and so on (all the yummy metadata stuff that forensics thrives on). I don't disagree with the latter, but I don't see the correlation with the former. The former is more the Incident Response (IR) type, it seems, and in my experience those folks are rather well versed and cognizant of maintaining evidence integrity (such as all that yummy metadata) and chain of custody. Pure IT folks - sysadmins and such - not so much; that's not to place blame, it's just not their area of expertise.

So here's my summary:
If your company is under regular litigation - large or small - and perhaps if you have regular threats to your intellectual property (thinking internal threats here, not external), it may well be a wise move to look at developing in-house capabilities. You need to really take some time to determine your internal needs and requirements, and remember these matters are about more than just email (systems, network, database, etc), and you must have a good grasp on your environment variables. You need to determine how much of the process you want internal - you may still want to outsource final production and hosting, for instance. Make sure you get the right expertise, and be aware that there will be an up-front cost (ongoing costs for software, hardware, training and certifications are minor in comparison). But the savings can be significant, and it is possible to come out ahead, if you compare against the money you would have been spending with outside vendors. ROI, the language of C-levels... :) Bottom line is, be informed, and make intelligent choices - don't just take action based on what either "side" is telling you.

I do think it may not be the best decision to try to convert your IT staff. In years of dealing with IT departments, and knowing how those personnel tend to think/approach these matters, your up front difficulties and costs are much higher, and you have a much steeper "learning curve," if you will. It pays to get someone who already knows how to do the work, has solid experience, and I would even add, has provided expert testimony in court. That is the bottom line for this field, whether one - and one's work - stands up well in court. But do be careful, as not all consultants are suited for corporate life; it's a different style of work, and you need someone for the long-term, not short-term, or your ROI decreases. You also don't want a "push button" forensics person, but someone who truly understands what's going on behind the scenes; they're going to be able to provide better development and support for your internal programs.

Let's face it folks, litigation isn't going away, nor is electronically stored information (ESI). Thus, ESI will have to be produced in litigation, and in comes eDiscovery. Orgs large or small feel the sting of the associated costs (which seem to be rather unreasonable at times), and - just being realistic - people are going to look for ways to bring it in-house. Sometimes that's just not feasible, and in those cases I think it's important to look for help in pre-culling to reduce costs. But for many organizations, having an internal program makes perfect sense and is not a mistake - when approached carefully and done right.

Okay, I think that's about it. No tech stuff this time, sorry to disappoint those who might've hoped otherwise.

Using Log2Timeline

Log2Timeline and forensic timeline creation


Creating timeline w/mmls, fls, log2timeline
This is really written with an image of a Windows system in mind. This was written as a guide for our lab (and me, to help remember), so keep in mind it's not necessarily intended to be a polished presentation. Some things were added/changed as my use of l2t progressed. The big difference between this and what has been published on the SANS blog and on Kristinn Gudjonsson's site is the use of 'find' and 'while' loops to recurse through directory structure instead of (for instance) going into each user profile for the ntuser.dat file.

These l2t cli bits of this are largely irrelevant now, with the introduction of the timescanner front-end (which does a great job of automating everything for you). However, there may (hopefully) be some other things of interest/use in different scenarios.

You do not have to extract the partition, but it's helpful to have a single image file for mounting purposes (I'm not sure about mounting a split image; at the least it will be easier to have a single/concatenated file)...

NOTE: Where something is enclosed such as [imagename] this includes full path to that file where needed; the "name" is a variable assigned by the analyst.

You need to gather partition info, which can be done with mmls or fdisk:

mmls [imagename] - this will show partition info (even from split image files), and you can identify whichever partition(s) you need (specifically you need the start point).
or
fdisk -ul [imagename]

To create the baseline bodyfile, use the following:
fls -m [original mountpoint, ie C:, /boot, etc] -r -f [file system type, type 'fls -f list' to find the values] -i raw -o [offset/start sector info from mmls] [path to imagefile] > [path to new bodyfile]
NOTE: The offset here is the straight info provided by mmls; it is not multiplied by 512 (sector info)

Variables here are:
C: - this is however you want it to show in the bodyfile for drive letter; keep in mind if you are working multiple partitions... (-m is what allows you to set this parameter)
ntfs - this will depend on the file system in question (-f denotes you will provide filesystem)
raw - this will depend on the image type (-i denotes you will provide image type)
offset info - as stated, this comes from mmls or fdisk (-o denotes you will provide offset)
recursive - this is denoted by (-r) which says you are recursing into all subdirectories

Bodyfiles are converted to timeline format w/mactime command (you don't do this now; it will happen at the end):
mactime -b [bodyfile] -d -m -y -z [timezone] 2000-01-31..2000-02-01 > [timelinefile]
Variables here are:
-m and -y: go together to establish date format - yyyy mm dd and day of week
-z: sets the timezone of the original system/image
-b: you will define the bodyfile to be used
-d: provides output in CSV format (this will add the timestamp to every line, even if it's the same second - you need this!)
dates: date range provided in yyyy-mm-dd..yyyy-mm-dd format - must use both, even for single day (in which case, the 2nd date MUST be one full day after) such as:2009-11-10..2009-11-11
NOTE: For original/main bodyfile, you won't use the date parameters; you want the entire system bodyfile intact (you will split out later if needed)
NOTE: There are some cases where you are having to process multiple bodyfiles into a single timelinefile, and want to use >> to append instead of overwrite. Be careful!

Creating log2timeline bodyfiles:
You can use the following to get lists of l2t variables/parameters:
log2timeline -f list

First you must mount the imageset loopback to be able to browse the filesystem for use w/l2t:
Multiply the Offset info (from mmls or fdisk) by the number of bytes per sector (also from mmls or fdisk), typically 512 (such as 63*512=32256)
mount -o ro,loop,offset=32256 -t auto [imagename] [mount point]
Variables here are:
offset: this will depend on the result of your partition math, as noted above

NOTE: I use mkdir to create my mountpoint, using things like "src" for source and "dst" for destination instead of relying on drive lettering (such as sdc1)

Now you can browse the mounted filesystem for your imageset, in order to create bodyfiles for l2t...

cd to any relevant profile, for ntuser.dat files
log2timeline -f userassist NTUser.dat > [UAbodyfile]
or from within Documents & Settings, do:
find . -iname ntuser.dat | while read d; do log2timeline -f userassist "$d" >> [UAbodyfile]; done ****This works better - more automated****

while within profile, run LNK files (if there are multiple users, do so within Documents & Settings):
find . -iname *.lnk | while read d; do log2timeline -f win_link "$d" >> [LNKbodyfile]; done

while within profile, run IE history (if there are multiple users, do so within Documents & Settings):
find . -iname index.dat | while read d; do log2timeline -f iehistory "$d" >> [IEbodyfile]; done

while within profile, run Firefox 3 history drill down into Application Data/Mozilla/Firefox/Profiles/xxxxxxxx.default and do:
log2timeline -f firefox3 places.sqlite > [FFbodyfile]
or from within Documents & Settings, do:
find . -iname places.sqlite | while read d; do log2timeline -f firefox3 "$d" >> [FFbodyfile]; done ****This works better - more automated****

cd to Recycler directory, and down into each S-1-5...
log2timeline -f recycler INFO2 > [RBbodyfile]
or from within RECYCLER directory, do:
find . -iname INFO2 | while read d; do log2timeline -f recycler "$d" >> [RBbodyfile]; done ****This works better - more automated****

cd to System Volume Information/_Restore/RP... for restore point info (you do not have to go into each subfolder)
log2timeline -f restore . >> [RPbodyfile]

cd to Windows/Prefetch directory for prefetch info
log2timeline -f prefetch . >> [PFbodyfile]

With all of your individual pieces, create a copy of your original system bodyfile, and add the l2t bodyfiles into it:
cp [bodyfile] [bodyfile2]
cat [XXbodyfile] >> [bodyfile2]
Repeat for each l2t bodyfile (denoted by the "XXbodyfile")

You now have a single bodyfile which contains all the l2t variables/optional bodyfile info. This is the point you will start splitting out timeline sections with date ranges:
mactime -b [bodyfile] -d -m -y -z [timezone] 2000-01-31..2000-02-01 > [timelinefile]

___________________

PS: I am very disappointed that Kristinn did not get the recognition he deserves through this year's Forensic4Cast awards. Log2Timeline is by far and away (in my book) the absolute best software contribution to digital forensics, and should have won that category hands down!

Computer Forensic Blog Intro

Computer forensics blog, 'forensicaliente' ~

So, the name is completely geeky, but hey, that should be worn like a badge of honor. Or perhaps a SANS Lethal Forensicator ROM (Round Metal Object, aka, 'coin')... :) No, I'm not sayin' for me, I'm just sayin'.

In looking for something in IT jobs that really truly interested me, I came across computer forensics (digital forensics, whatever you want to call it), and managed by the grace of God to get into the field. I have been working here for several years now and absolutely love it. I think it's the coolest thing since sliced bread, pockets on shirts, and so on. Or with slang/euphemisms being what they are, sometimes 'cool' = 'hot' and thus 'forensicaliente...'

As I work through problems, come up with questions to test and try, I have developed a fairly sizable Evernote library (or maybe not so sizable, who knows?). I finally decided to put things into a blog in the hopes that they may help others. That's one of the things I think is so neat about CF, that so many practitioners share so generously, and I'd like to contribute to the community as well.

While I won't likely be as prolific as some of the great forensic bloggers, I will try to start out with a number of posts from my library of notes, and add to it as I go along. I hope these are of use and benefit to others.


PS: And yes, the site is largely unformatted; I'll be working on that too.