Did You Know? ... or ... What Is Normal?

We all know that in Windows, explorer.exe (the user shell, graphical file system interface) is the parent process to applications launched by the user, such as Internet Explorer, (iexplore.exe).  That's normal; we all know it, and it looks a little something like this:

That's great and all, but did you know that's not always the case, at least in the matter of iexplore.exe?  Sure, of course you did.  Maybe you haven't thought about it before, but you do know.

What happens when you're on a 64-bit system, and launch the 64-bit version of Internet Explorer?  Did you note the drop-down arrow next to it in the screenshot?  Well, that looks a little something like this:

First, there's the "parent" iexplore.exe (pay no attention to the changing PIDs, please).  Path is "Program Files."  But then the "child" iexplore.exe; path is "Program Files (x86)," indicating this is a 32-bit spawn of (well, you know...) which thus means that the parent is 64-bit.  So, a "new" "normal," then?

   

                  

And of course you realize that if IE is the default browser, and hasn't been launched when you click a link in an email application (such as Outlook), then Outlook will be the parent of IE, rather than Explorer.  Another normal.  You get it.  Here's how that pans out:

So, the user clicks a link in email (outlook.exe), which then spawns the browser (iexplore.exe).  Just as I noted above.  What's cool here is that the command-line parameters (on the right in the screenshot) show where the user was being taken.  Good stuff.

So anyway, the other day I thought I'd do some digging into all the times that Explorer is NOT the parent of IE; partly I wanted to challenge my knowledge, but also to see if I had an opportunity to find any evil, or build query filters that would help separate the signal from the noise for evil in the future.  I ran some queries to find all instances of IE in my environment, where Explorer was NOT the parent process.  There are actually quite a few, you might be surprised.  The predominant ones were:  iexplore.exe, svchost.exe, and outlook.exe.  Okay, we've already discussed the first and last of those, but the middle?  Do what?

First, let's revisit the first one, because this is not the same as above; this is 32-bit on 32-bit action at its tabular best:

Parent process is on the left, child on the right.  Then on the right in the middle, you have the SCODEF and CREDAT references, which indicate an IE tab.  SCODEF points to the PID of the parent.  If you look back up at the ProcessHacker screenshots above, you'll see the parent IE is PID 7760; this is referenced by SCODEF for the child process.  And it's not just me, covered in Cheetos (R) dust, making this stuff up.  Here's a reference (granted, for IE8) from MSDN Blogs. 

SVCHOST!  SVCHOST!  WE WANT SVCHOST!  

Yes, yes, I promised you svchost.exe as a parent to IE.  Now, let's pause for a second and remind ourselves what is normal for svchost.exe, too.  While there may be multiple instances of svchost.exe running at any one time, the parent will always be services.exe.  That's normal, and we all know it.  Okay, remember that.  There is a scenario, which if you've spent some time reviewing the SANS DFIR Find Evil poster you are aware of, wherein IE can be started via the command-line "-embedding" parameter; in this instance, the parent won't be explorer.exe.  Looks like this:

That -embedding switch is over on the right, and as you see on the left, the parent is svchost.exe.  Done, right?  No, not quite yet.  Take a peak at what's next...

What we're looking at here is just a wee bit different.  This isn't so much about IE, although that's come into play, too, but more to the point of svchost.exe, and it's parent.  If you see a parent other than services.exe, you may start getting concerned about malware.  Then you see rpcnet.exe, which sounds legit (ish), but still isn't normal, and you're probably more concerned about malware, since malware often uses names similar to legit names, so as to look "normal."  In addition, this rpcnet.exe is signed, and we all know that signed code is used to bypass detections in antivirus, HIPS, and other products.  So, is this malware?  

Well, opinions might vary, and it certainly behaves like malware.  However, it is - in this instance - normal and legit.  It's actually associated with embedded tracking software to help deal with stolen computer assets.  Of course, while I might "know" that, someone could be trying to get one over on me by masquerading their malware as a process I'd expect to see, so how could I further verify that?  If you know anything about this tracking software, it's not designed to be "normal" and is difficult to validate - it really truly does operate very much like malware.  So I'd most likely have to turn to other sources of evidence, possibly even packet captures, to see where it was going and how it was communicating.  If you want to know more about why validating your findings and being able to do so from multiple types/sources of evidence, come to my talk (To Silo, or Not to Silo: That is the Question) at the SANS DFIR Summit in Austin this June.  

Anyway, the SANS DFIR Find Evil poster talks about knowing what "abnormal" is, but in order to know that, you have to know what "normal" is.  Old story, but that's the same way people are trained to spot counterfeit money - know what "good" money looks like, to be able to spot what's not.  When it comes to normal with computers, and especially in enterprises, there are "global" norms and "environmental" norms.  The globals are things like the 32-bit spawn from the 64-bit parent IE, the SCODEF references for child tabs (which includes the home page, by the way), and Outlook links spawning instances of IE to reach the websites.  Environmentals aren't out to save the computer, but are things like tracking software sitting in between services.exe and svchost.exe.  If you know what those are for your world, you'll be much better off when it comes to finding evil, and separating the signal from the noise.

Happy Hunting!

Presenting DFIR, Shakespeare Style - DFIR Summit 2014

I have been given the opportunity to speak at the SANS DFIR Summit in Austin this year, on a topic that I think is very important.  That is, whether there is value in focusing on one discipline within the DFIR realm - not only from a skillset perspective, but also during investigations.

You can read more about the Summit on the SANS website, but here's a quick overview of my talk (titled To Silo, or Not to Silo: That is The Question):

Have you ever heard someone say they do network forensics and don't need a host computer to know what happened (or vice versa)? Or an incident handler analyzing RAM make a comment about disk imaging being unnecessary and outdated? Unfortunately, these types of mindsets are problematic because they are limiting - to the investigator, to the evidence, and to our profession.

These limitations show themselves through incomplete analysis and inaccurate conclusions. If the limitation is real, tangible – for instance if firewall logs are the only available evidence – then we make the most of what we have. Otherwise, incident response should be based on all of the information available to us as investigators – firewall logs, packet captures, system alerts, RAM, filesystems, malicious executables, and so forth. If these are available, but are ignored or overlooked, analysts are missing out on potentially valuable information. When that happens, the conclusions drawn and recommendations made will be incomplete or just plain wrong. In the words of Hamlet, "Ay, there's the rub."

In this presentation, the audience will be taken through several different real-world scenarios dealing with potentially infected systems, where pieces of evidence are available from some of our "competing" disciplines. Background on each system will be given, to include how it showed up on the radar as potentially compromised; again, this stresses the point that we don't know what happened until we examine all of the available evidence. With each system, different types of evidence or DFIR disciplines are available to help with analysis; these examples will show how each - by itself - falls short in painting the full picture of what happened, and will illustrate our inability to draw concrete conclusions without all the pieces of the puzzle.  Without being exhaustive, this presentation will demonstrate the importance of having knowledge, skills, and abilities in multiple DFIR disciplines, and how looking for additional evidence sources can help us perform more accurate analysis and reach more accurate conclusions.

PS:  SANS has a $1000 discount when using the code "SUMMIT" - this is available from March 17th - March 31st.  More info available here.

Check Your Hash at the Door

A friend recently asked if I would post about using md5deep; he was trying to figure out how to use it to compare against the NSRL hash library and was having some difficulties.  Well, maybe it's stretching to say he asked if I would.  He asked for some help in figuring it out, and at the end of it all, I offered to post some stuff; he said it might be useful.  So if I put a PR spin on it, I was specifically asked, because of my great and wondrous knowledge.  How's that?  :)

Oh, and if you're wondering at my use of the term "friend," you can always substitute "person who pretends not to hate me as much as others do" if that makes you feel better.  I know some of you don't think it's possible that I actually have friends, so I feel I have to give you a way out...

Anyhow, to be honest, I've not ever used md5deep for that specific purpose.  I've always used something like FTK or EnCase.  I know, I know, don't be hatin'.  I do a lot with command line tools, and prefer them in many - if not most - cases, but there are some instances where (completely dependent on the scenario), a GUI is just easier and/or the better option.  And for what I've done with the NSRL hash libraries, those tools were what was called for (there was more than just deNISTing at hand).  

That said, the way I've used md5deep should still accommodate comparing against the NSRL hashes, with just a little bit of tweaking.  I've always used it to compare and verify files being copied from various data sources (typically network shares), when using a copy application that couldn't do so.  So in that scenario, it works out something like this:

We want recursion, possibly a progress/status indicator, and an output file that can be used as the input file for hash comparison.

Hash the source like:

user> md5deep -re "g:\loose_PST_files" > f:\psthash.txt

Hash the destination (and compare to hash of source):

user> md5deep -renx f:\psthash.txt "f:\loose_PST_files" > f:\psterrorhash.txt

This breaks down as follows:
    -r = recursion
    -e = progress/status
    -n = displays names of files not matching the input file
    -x = negative hash matching; reports files not in the input file 

    g:  is the mapped or local source drive

    f:  is the destination drive
  
Order for source is:  > # the flags have to be combined together (ie, -re) or it errors.  It will automatically output to stdout, so we have to redirect to logfile; this gives filename and hash value only (other format cannot be read properly for use as input file).

Order for destination is:  >   # the error log immediately follows the flag as the first variable; this is read and compared against the hashes of the destination.  Any errors in the comparison (whether the file is there but shouldn't be, or isn't there but should be) will output to the error log.  The size of the log should be 0 KB when all is done, and (obviously) empty - thus showing there were no errors/mismatches. 

These were notes I made in the past, and I think were most recently updated with version 3.4.  The tool is now on 4.3, so I don't know if the following has changed, but ... as noted above, the md5deep comparison (input) file had to have hash value and filename only, or could not be properly read.  I couldn't find anything in the documentation to say one way or the other, but it's possible that the NSRL lists might be problematic, even though md5deep is only looking at the hash and not the filename/location.  I did have occasion recently to verify the integrity of some evidence migrated to new storage, and ran md5deep in the manner above; it worked great.  The only thing I'll note is that if you do want the progress indicator, it WILL slow down the process; use with discretion.  You have been warned.

There were a couple posts related to using NSRLSrvr and NSRLLookup that showed up on the radar.  The first I passed along, and the second my friend found on his own; apparently it was helpful.  I don't know the first author, but Patrick puts out some good stuff on Sysforensics.  If you don't keep tabs on his blog, you should.  So without further ado, here they are, and they should work just fine to use md5deep to compare against NSRL.

http://blog.jameswebb.me/2013/05/setting-up-forensic-hash-server-using.html

http://sysforensics.org/2013/12/build-your-own-nsrl-server.html

And this wouldn't be complete if I didn't mention the NSRLookup service from Kyrus Tech:

http://www.kyrus-tech.com/nsrlookup-service-beta/

Which I first saw mentioned last year on Jesse Kornblum's site:

http://jessekornblum.livejournal.com/278435.html?nojs=1

Have fun and happy hunting (or, uh, happy hashing)!

What's Hash Got To Do With It? (Part 2)

Just a quick follow-up to yesterday's post, as promised.  I decided to do it as a new post rather than an edit/update, figuring it would be easier to track this way.  Not much to it, just a little more info to round things out.

First, I got my hands on the infected PC today, pulled the hard drive, and extracted some files.  I wasn't able get RAM, but I do have hiberfil and pagefile to work with, along with the user profile and registry hives, but that's not the point here.  (I will note, however, that unless I can find evidence of the malware's activities in hiberfil or pagefile, I'm still - to an extent - surmising how it spawned msiexec.exe in the first place.)  I found three separate executables under the user profile, as created by msiexec.exe.  I copied them off into a separate directory and used md5deep to hash the files, so that I could look for them on VT or Malwr.com as well (there, now this is a post about md5deep too!).  Here's what I ended up with (my own formatting):

26e57bde90b43cf6dae6fd5731954c61 | msevaxlgn.exe  
26e57bde90b43cf6dae6fd5731954c61 | mssuhin.exe
26e57bde90b43cf6dae6fd5731954c61 | msyaam.exe

Did you notice that?  All hashed the same.  Okay, maybe that's expected, or not too unexpected at least.  But did anyone note the hash value from yesterday, for the initial executable (from inside the zip)?  Well, in case not, here it is again (in the same format):

26e57bde90b43cf6dae6fd5731954c61 | order_jd4320480293.exe

See that?  Yes, that's right - another md5 match, and I doubt it's due to collisions.  :-)  So basically, the original malware spawns msiexec.exe, creates a new executable(s) that is (or are) identical to the original (except for the name); the original is deleted, and the show goes on.  I don't know about you, but that's not what I commonly see from malware.

Now for a few more tidbits...

A little header info from a network capture:

POST /se/gate.php HTTP/1.1
Cache-Control: no-cache
Connection: close
Pragma: no-cache
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/4.0
Content-Length: 74
Host: finley.su

IP Addresses, as seen by the C2 blocking device:

Now, I mentioned before that there was a .ru TLD involved, and here that is:

Haven't looked at that traffic yet, and it never made it to the device that blocked the C2 traffic, as it had already been blocked for other reasons.  And of course, until I examine the pcaps, I won't know what that traffic was (still might not - since it was blocked, it will be a one-side conversation).

Last of all, a screenshot of the offending email (note some inconsistencies outlined in red):

SupportAmazonyu?  AmazonStoreIdecoa?  I could get it with "Amazon," but what's with the other stuff?  And the order numbers don't match up anywhere.  And finally, it says, "Well let ..." rather than "We'll let ..."

Well, I guess you can't patch users!  At least not without regression testing first.  ;-)

I think that's it for now; if I do come up with anything else that looks interesting or useful, I'll post another update.  Aside from that, I told a friend I'd post about some specific uses for md5deep; we'll see how that works out, but hopefully it will be coming soon.

Happy Hunting!

What's Hash Got To Do With It?

Well, because that's how we're going to kick off this little story.  Without further ado, here ya go:  26E57BDE90B43CF6DAE6FD5731954C61

Before we go too far into that, however, I need to take a step back.  It's been a long time since my last post.  Too long indeed (my job makes it difficult, to say the least), so let's not dwell on that, shall we?  With that out of the way, I'll give just a quick back story on the above-referenced md5 hash value.  Last week, on Wednesday and Thursday, I came across this malicious gem.  Well, I think it was the same both times.  From a network perspective, it was behaving the same - reaching out to the same C2 channels, attempting to post the same type of encrypted data (based on format, appearance of cipher-text, etc.).  However, as it turns out there wasn't an appropriate endpoint sensor on the first box, to be able to pull back more relevant info.

That fact is important as we go on - that unless you have evidence across different DFIR disciplines (host and network, with all their subsets), you really can't say for certain what is occurring.  You can postulate, and may very well be correct, but without RAM, RE, timeline and other host data, and various points of NSM (network security monitoring) such as FW logs, IDS/IPS, netflow, you're left at least somewhat in the dark.  Anyhow, from a network perspective, these two appeared to be the same, and fortunately there was an endpoint sensor on the second box; I got to have some fun, and thought I'd share.

As it turns out, the infections originated not from some stealthy drive-by or infected website, but via an executable inside a zip file sent along with an "Amazon" email to personal email accounts.  In the case of the second one, the user had received two previous ones that were ignored and deleted; the third was just too much and curiosity won out.  User opens zip file.  Oops #1.  User double-clicks executable.  Oops #2.  At that point it's all over but the tears, and the malware has its fun.  The positive side was that - finally - I got to have some fun too!

Now, I didn't know all that at first.  All I knew was that C2 traffic to finley.su had been blocked.  That was the same domain as the previous day, so my interest was piqued.  The system that blocked didn't have any info about the malware, just knew that the traffic was bad, so that was no help.  I went in to my sensor logs from that box, and found the application reaching out to finley.su was msiexec.exe, from the c:\windows\system32 directory.  Ooh, now that's not right.  The parent process for msiexec.exe was order_jd4320480293.exe.  Now that's not good either.  And it was running from inside a zip under the user's temp directory, with a parent of explorer.exe.  Alright, that's not unexpected, especially at this point.

I was able to pull a copy of the binary for order_jd4320480293.exe that the sensor had pushed to my analysis server, and saw that it had 9 hits by hash on Virus Total, so I went to check that out.  There were indeed 9 engines that detected it; one from the 11th, and eight from the 12th.  Most had generic names as yet; the most relevant one appeared to be from McAfee, which had it as a ZBot variant.  Searching on that lead me to Malwr.com, where the binary had apparently been uploaded and analyzed already.  As a side note, the next morning VT had 14 detections, and is now up to 23 (What's funny is that some of the vendors I submitted a sample to, now say they detected it back then.  Hmm, really?).  Since there's some RE type info on VT and Malwr, I won't try to go into that; I'll give you some reference links to go enjoy that.  I will, however, give a quick rundown of what I saw.  I will probably come back and add some IP addresses for the finley.su domain, as well as a .ru TLD I saw from the endpoint, so watch for an update.

So here's the quick high-level rundown:

*  User downloads zip from fake Amazon email, opens, and runs order_jd4320480293.exe

*  Malware executes, and order_jd4320480293.exe spawns (loads) msiexec.exe from c:\windows\system32 as a child process 

*  Child process msiexec.exe creates a new executable under the user profile, named (one example) msyaam.exe or msevaxlgn.exe (another example)

*  Child process msiexec.exe adds a runkey for the new executable, in the user hive, under Software\Microsoft\WindowsNT\Current Version\Load

*  Child process msiexec.exe deletes original executable, order_jd4320480293.exe

Along in there somewhere, msiexec.exe packages up some data (probably credentials), encrypts, and tries to ship it out to remote servers.  Unsuccessfully.  Defense in depth... ;-)

Well, that's it for now.  Given the info available elsewhere about this malware, I probably won't post anything about its actions on the system, unless I see a very different  approach take place.  But I will look to post some IPs and perhaps some other tidbits that might be helpful from an IOC perspective.  Other than that, here are some links:

Virus Total:  https://www.virustotal.com/en/file/00fb48cae855a1321e9edcd2b0d9a4935dc7954f0221030a3b01dbdfe0332a44/analysis/ 

Malwr:  https://malwr.com/analysis/MDU2ODNhYzA1MmIzNGZiYTkxNmQ5OWJmNGM0OWFiNGU/#static

Sophos:  http://www.sophos.com/en-us/threat-center/threat-analyses/viruses-and-spyware/Troj~PWSZbot-X/detailed-analysis.aspx

Happy Hunting!

Getting Your Feet Wet / Joining the Conversation

Well, I've been "working" on this post for quite some time now, and just haven't ever wrapped it up.  Bah.  To show how long I've been "working" on it, I came up with the idea before Richard Bejtlich posted on the Mandiant blog about InfoSec career building and before Chris Pogue posted about his job change (and where he hinted at the possibility of a "Sniper Forensics" book - bring it, Chris!), or about careers (his 2nd post on this topic).

Part of this comes from a comment that Hal Pomeranz made a while back.  He said (general paraphrase) that there's no better use for social media than to help others.  In context, he was talking about InfoSec jobs.  Hal's a guy that I highly respect, and since I've witnessed - first hand - his willingness to put his money where his mouth is (so to speak) in this area, I take it to heart.  By the way, he has an excellent series on working for yourself over on his blog (keyword: consulting).

A number of great folks have posted career-focused info, including those above, and it's more recently entered my radar as I'm in more of a position to help others.  I don't have the "street creds" they do, but I wanted to offer up a few things I've put together.  As I moved late last year into managing our InfoSec group, as well as heading up the IR team, I've had the opportunity to mentor a couple of newcomers to our field, and I put this together in part for them, to give them some additional resources.  I highly agree with what others have said, that putting yourself out there is important - blogging, tweeting, mailing lists - just talking and sharing with others.  I won't go into that in any depth, as I think it's been very well-covered elsewhere; I'll just re-emphasize that it's important.  I've seen it myself, where potential employers check out blogs, activity on email lists, and so on; it definitely makes a difference, because hiring someone in this field revolves around having confidence that they KNOW what they're doing, and can DO the work.

So with that said, if you're new to InfoSec (security, forensics, incident response, auditing, etc) here are some resources that can help you start to get more comfortable and plugged in to the community.  And it IS a community, more so than many other fields I've seen. 

Mailing lists:
Dragon News Bytes
PaulDotCom
Win4n6

Websites:
DFIR Online - this is an excellent resource, and also hosts the monthly "DFIR Online"
PaulDotCom - yes, it's showing up again  :-)
SecurityFocus
Darknet
CommandLineKungFu – this is just awesome and hilarious too
HolisticInfoSec - Russ has some great tool writeups
KrebsOnSecurity - Great resource on cybercrime
Team Cymru
US Cert
SANS Reading Room
Internet Storm Center
Jesse Kornblum
Lenny Zeltser
SANS Computer Forensics
ForensicArtifacts
A Fistful of Dongles
Hacking Exposed

Books:
The Basics of Information Security
Hackers Beware – older, but very good info
Network Security Bible – another one by Dr. Eric Cole
DFWOST
Hacking Exposed - any of the "Hacking Exposed" series
WFAT - any of Harlan Carvey's books
Practical Packet Analysis
Violent Python
I’ve found great deals on books at Half-Price Books, which can make a big difference.  Some of the older ones, you might be able to find at the library as well.

Twitter:
I really recommend you get on twitter if you’re not.  Have a profile that’s focused on what you’re interested in, and follow people in that field.  It can be a great source of information, as well as connections when you need to know something.  Here are just a few folks that may be good to start with:
Johannes Ullrich
Josh Wright
Lance Spitzner
Russ McRee
Wesley McGrew
Doug Burks
Christiaan Beek
Eric Cole
Brian Krebs
Mike Cloppert
Richard Bejtlich
David Cowen
Didier Stevens
Lenny Zeltser
Hal Pomeranz
Chad Tilbury
SANS Forensics
Rob Lee
Andrew Case
See who they're talking to, and start branching out with who you follow.  Don't be afraid to join a conversation, ask questions, and share your experiences.  There are also quite a few active DFIR types on Google+, and there have been some good conversations happen there (at more than 140 characters a pop), as well as some hangouts.

Hope you find it helpful.

PS:  I have been advised by Counsel to at least mention that this list of resources is by no means exhaustive, nor intended to be.  In addition, they are in no particular order, nor intended to be any sort of status qualifier, and I'm not getting paid in any way for these references (aka, name dropping).  They are just some of the resources I find helpful, and wanted to share.  If you , your site, your book, or your list are not mentioned, that doesn't mean I don't follow, read, etc (see the whole "not exhaustive list" piece).  There are several hundred folks I follow on twitter, over a hundred blogs, dozens of books, and websites galore where I gather info while on this journey.  Quite simply, too many to mention.  Thanks to you all for being available and sharing with the community!