Sweet Child o' LSASS

Recently, I was channeling my inner rock star, and thought I'd share a finding regarding "normal" occurrences.  You're probably all familiar with LSASS.exe, the "Local Security Authentication Subsystem Server" process, and you might also know that it doesn't have any children.  Poor thing; children are truly a gift (and a challenge, but that's a different topic).  Anyhow, as noted in the SANS DFIR "Find Evil" poster, if LSASS spawns a child process, it bears looking into - and that's exactly what I was doing.  

Given that I think it's important to be as proactive as possible with regard to incident response, I am always looking for ways to spot potential problems.  Now, the SANS poster showcases things to watch for when doing memory analysis, but if you're parsing all executable activity in real time and storing that data in a way it can be queried at will (kind of like Sysinternals procmon on steroids), then why not apply the same principle and see what can be found?  Yes, yes, I'm talking about CarbonBlack (now part of Bit9), which is (in my opinion) an awesome endpoint monitoring platform.  However, while this post will make use of that technology, don't think of it as being about Cb, but rather about the hunt, what's found, and how that informs the bigger picture (and may change some of what's considered "normal").  Keep in mind there are other tools that can help accomplish the same goal, and as noted, memory analysis (with tools like Volatility, Rekall, and Mandiant Redline) is at the forefront - so don't get hung up on the tool; it's the investigator that makes the difference in the long run. 

With that, I'd like to tell you a story about the hunt for spawn of LSASS, and how it started with a simple little query, as shown below; basically, any process for which LSASS.exe showed up as the parent...

Right away we see that there's one sweet child o' lsass, on 37 endpoints, with two different hashes showing up.  Okay, so two different binaries or versions, then.  Let's keep digging, starting with a listing of the search results.

A couple things stand out; namely that each process is associated with six (6) filemods (such as create/modify/delete a file) and two (2) netconns (could be browsing to a website, IP address, hostname), while related activities for registry (regmods) and other processes/binaries (modloads) are similar in count but different.  If you're the type (and I am) that likes to review data offline to filter, sort, search, and so on, you can download a CSV that looks a little something like what's below.  If you have oddball md5 values, abnormal paths, or process names that stand out, it's sometimes easier to focus in on (at least for me).  With this, the "start" value is the time the current instance of the process ran, and "last_update" is the most recent time it actually did something as it applies to the query at hand.

I mentioned oddball md5 values, right?  And we know we have two different ones at play here, and 37 total processes, so what's the breakdown?  Funny you should ask (well, I asked, but it was kind of rhetorical anyway - just work with me...)

So apparently two out of the 37 are the "f17e" hashes, with the remainder being "bcb8."  And yes, that can be identified using the GUI, but I like to see things with my own two eyes and plus this is an offline record in case I ever need it.  No abnormal paths were noted.

Anyway, since there's consistency with filemods and netconns, that's a good place to start looking, but first, I'd like to know a little more - high level - about these processes.  From a tool standpoint, Cb has a "preview" feature, and so to take a peek at the different binaries involved here (remember, two different hash values)...

First we had "Mr. Popularity," the "bcb8" version, followed up by the indomitable "f17e."  In either case, the command line parameters are the same:  "/efs /enroll /setkey."  If you had not already, you're probably running out and searching teh intarwebs for this executable, EFS, and whatever else might give some insight, since it appears to be from Microsoft (R) and might be legit (but you never know, right?).  If that's what you're doing, no worries, I was in the same boat.  I even reached out to the SANS DFIR email list to see if anyone else had encountered this, since all know that "normal" means LSASS doesn't have kids.  No children.  Nada.  Zip.  Zilch.  Carlos Marins pointed me to the following MS document (link is direct download) http://download.microsoft.com/download/9/8/2/98207DD4-7D2C-4EF6-9A9F-149C179D053E/CommercialOSSecFunReqsPublicV2Mar09.docx, which was very helpful in understanding some of the things I would subsequently find.  Chris Pogue asked a few questions and reminded me about checking the hash with Bit9 File Advisor.  

Speaking (er, well, writing) of hash, we'll go ahead and knock that out.  VirusTotal and File Advisor both came up clean:

Alright, now back to the fun stuff - filemods and netconns for this sweet child o' lsass.  Oh, first - this'll be quick, I promise - we can take a look at more details about the two binaries in question (as binaries, not as the named processes).

In addition to a few more details, this will show how many times that particular binary has been seen in action, without any time or other filtration (such as by parent process).  In addition, using the aptly-labeled "Download" button, I can extract a copy of the raw binary for additional analysis, reverse-engineering, and so forth, offline.  More on that later, as I said this would be quick.  Now, back to the rest!

Here's a quick look at each of the processes for analysis:

You can see that the relationship between wininit.exe and lsass.exe is as expected.  It's just that the latter spawns efsui.exe as a process (which of course we already knew by this time).  We see the commandline parameters again, and the fact that it's signed by MS.  What's new here is that the username (obfuscated) is the domain account of the actual end-user; it's not System or other non-human (actually, it's in the binary preview window as well, but I really just wanted to call it out here, rather than there).  Also, pay attention to the "Export events to CSV" button in the top right (more on that later as well).  And, more of the same from the other version of the process...

We were going to look into filemods and netconns if I recall correctly (and of course I do), so don't stop now...

You can't see the whole screen above, but just underneath the process map, there are some "facets" to speed up queries/drills/searches based on different criteria (of course, we're going to look at filemods and netconns - aren't you just tired of hearing about how we're "going" to do that?).  

Clicking on filemods highlights some other areas within that category of activity, and also focuses our results on just those.  Thus, we can see that the actions are broken down equally between creation and first write, and that only three directories were utilized, all within the user profile under AppData\Roaming (clicking on any of those would highlight only those pieces of information, thus narrowing the search further).  Next up are the event details.

Here, in timeline fashion, we get to see file creation, first write (if there were any file deletes, we would see those too), and some details associated with each.  The expanding arrow (as shown on the top entry) shows frequency information (singular in this instance because of the nature of the path), and the "Search" (in blue) will take us to those results (for instance if there were multiple systems instead of just one, we'd get to see what those all were).  The "Search" box at the top right of the event list allows us to find any search string in the results (such as username, filename, part of the path, etc), if we had something we wanted to jump to quickly (or even just see if it was present).  What's of interest here are the paths involved, which start to make sense in light of the MS document I mentioned earlier (same for regmods, but I'm not going to go into those for the sake of brevity).

Clicking on filemod in the facets again clears that drill, and we can switch over to netconns.

Basically, the two netconns were pointing to domain controllers, and as can be seen from the frequency information, those DCs are quite common from a connectivity perspective (as one would expect, being DCs).  The interesting thing here are the ports involved, which makes it look like LDAP is involved.  

Okay, we're getting down to the wrap-up stage (thanks for sticking with me this far, I know it's been a lengthy post/novella at this point).  What it appears we have are known, signed, MS executables associated with the Encrypting File System (EFS) for transparent file encryption, reaching out as the logged-in user to domain controllers for authentication, using LDAP.  But is that really truly what's going on?  Do we know enough to say that at this point?  Are there any additional checks we could do, ways to validate/verify this theory?  

What about looking at other activity on the system for suspicious processes, network communications, or known malware?  Any evidence to suggest that EFSUI.exe had been injected with other code after it started running?  What about packet captures for these LDAP connections - are they really normal authentication for the process?  Any corroborating logs from firewalls, the DCs, etc?  Is EFS used in the environment, or is the user known to do so specifically?  What about the binaries involved - does reverse engineering (RE) indicate any oddities or abnormalities?  

Do you remember that there is the ability to extract a binary for offline analysis?  So RE is definitely a possibility.  Firewall and DC event logs should be reasonable to expect (although not necessarily a given).  Packet captures from the time of the event (in the past) would require some dedicated NSM for streaming pcaps, but would be a really good way to help determine what's going on inside those netconns.  And we can certainly dig into the more detailed process and binary analysis on the hosts in question.  That "Export events to CSV" button I mentioned earlier?  Gives output like this, with a CSV for each "type" of activity, and will also include child processes if available (there weren't any for efsui):

Note:  The "Summary" text file gives some info about the process or binary being analyzed (name, hash, path, etc).

These spreadsheets provide a veritable plethora of information about the process or binary being analyzed.  Don't know if you noticed, but there was a section on the analysis page, referencing "Alliance Feeds" - this provides info about matches to Virus Total, known bad domains, and other "intel" related to activity that might be full of evil.  Rather than a specific process to search for, you could also start with a given endpoint/host, any of these threat feeds, or some custom query for known indicators (based on firewall, IPS, or other threat "intel" you have).  Anyway, all that to say that if you wanted to correlate the activity from EFSUI.exe as a child process of LSASS.exe, to any other activity to help determine whether this benign or the most evil thing on the planet, there's a lot you can do.  And again, not just with CarbonBlack - you can set up a packet capture for a period of time (host, router, switch, firewall, etc), perform memory analysis, targeted triage (such as popularized by Harlan Carvey, Corey Harrell, and Chris Pogue to name a few key folks), or even (gasp!) a full disk image with timeline and analysis galore! 

Maybe the information here is enough to make a determination and speak authoritatively about what happened, and whether there are any unremediated or ongoing risks.  However, it's really ultimately about those risks, and as investigators we may not be the decision-makers (in fact, most likely are not).  We can inform the decision-makers of our findings and recommendations, but we also have to be honest and explain what options are available, what those options would provide (and at what cost), and what potential risks could be incurred (and the probability thereof) by not pursuing the aforementioned options.  Want to know more on this subject?  Come to my talk at the SANS DFIR Summit in June - To Silo, or Not to Silo: That is the Question.  More info is available about the Summit here.

Thanks again for "listening" to my tale about the sweet child o' lsass, and remember ... you may know that lsass doesn't have any child processes, but if you don't verify or validate that, you might just reach the wrong conclusions, and that probably wouldn't be good in a "real life" scenario. 

Happy Hunting!

Did You Know? ... or ... What Is Normal?

We all know that in Windows, explorer.exe (the user shell, graphical file system interface) is the parent process to applications launched by the user, such as Internet Explorer, (iexplore.exe).  That's normal; we all know it, and it looks a little something like this:

That's great and all, but did you know that's not always the case, at least in the matter of iexplore.exe?  Sure, of course you did.  Maybe you haven't thought about it before, but you do know.

What happens when you're on a 64-bit system, and launch the 64-bit version of Internet Explorer?  Did you note the drop-down arrow next to it in the screenshot?  Well, that looks a little something like this:

First, there's the "parent" iexplore.exe (pay no attention to the changing PIDs, please).  Path is "Program Files."  But then the "child" iexplore.exe; path is "Program Files (x86)," indicating this is a 32-bit spawn of (well, you know...) which thus means that the parent is 64-bit.  So, a "new" "normal," then?

   

                  

And of course you realize that if IE is the default browser, and hasn't been launched when you click a link in an email application (such as Outlook), then Outlook will be the parent of IE, rather than Explorer.  Another normal.  You get it.  Here's how that pans out:

So, the user clicks a link in email (outlook.exe), which then spawns the browser (iexplore.exe).  Just as I noted above.  What's cool here is that the command-line parameters (on the right in the screenshot) show where the user was being taken.  Good stuff.

So anyway, the other day I thought I'd do some digging into all the times that Explorer is NOT the parent of IE; partly I wanted to challenge my knowledge, but also to see if I had an opportunity to find any evil, or build query filters that would help separate the signal from the noise for evil in the future.  I ran some queries to find all instances of IE in my environment, where Explorer was NOT the parent process.  There are actually quite a few, you might be surprised.  The predominant ones were:  iexplore.exe, svchost.exe, and outlook.exe.  Okay, we've already discussed the first and last of those, but the middle?  Do what?

First, let's revisit the first one, because this is not the same as above; this is 32-bit on 32-bit action at its tabular best:

Parent process is on the left, child on the right.  Then on the right in the middle, you have the SCODEF and CREDAT references, which indicate an IE tab.  SCODEF points to the PID of the parent.  If you look back up at the ProcessHacker screenshots above, you'll see the parent IE is PID 7760; this is referenced by SCODEF for the child process.  And it's not just me, covered in Cheetos (R) dust, making this stuff up.  Here's a reference (granted, for IE8) from MSDN Blogs. 

SVCHOST!  SVCHOST!  WE WANT SVCHOST!  

Yes, yes, I promised you svchost.exe as a parent to IE.  Now, let's pause for a second and remind ourselves what is normal for svchost.exe, too.  While there may be multiple instances of svchost.exe running at any one time, the parent will always be services.exe.  That's normal, and we all know it.  Okay, remember that.  There is a scenario, which if you've spent some time reviewing the SANS DFIR Find Evil poster you are aware of, wherein IE can be started via the command-line "-embedding" parameter; in this instance, the parent won't be explorer.exe.  Looks like this:

That -embedding switch is over on the right, and as you see on the left, the parent is svchost.exe.  Done, right?  No, not quite yet.  Take a peak at what's next...

What we're looking at here is just a wee bit different.  This isn't so much about IE, although that's come into play, too, but more to the point of svchost.exe, and it's parent.  If you see a parent other than services.exe, you may start getting concerned about malware.  Then you see rpcnet.exe, which sounds legit (ish), but still isn't normal, and you're probably more concerned about malware, since malware often uses names similar to legit names, so as to look "normal."  In addition, this rpcnet.exe is signed, and we all know that signed code is used to bypass detections in antivirus, HIPS, and other products.  So, is this malware?  

Well, opinions might vary, and it certainly behaves like malware.  However, it is - in this instance - normal and legit.  It's actually associated with embedded tracking software to help deal with stolen computer assets.  Of course, while I might "know" that, someone could be trying to get one over on me by masquerading their malware as a process I'd expect to see, so how could I further verify that?  If you know anything about this tracking software, it's not designed to be "normal" and is difficult to validate - it really truly does operate very much like malware.  So I'd most likely have to turn to other sources of evidence, possibly even packet captures, to see where it was going and how it was communicating.  If you want to know more about why validating your findings and being able to do so from multiple types/sources of evidence, come to my talk (To Silo, or Not to Silo: That is the Question) at the SANS DFIR Summit in Austin this June.  

Anyway, the SANS DFIR Find Evil poster talks about knowing what "abnormal" is, but in order to know that, you have to know what "normal" is.  Old story, but that's the same way people are trained to spot counterfeit money - know what "good" money looks like, to be able to spot what's not.  When it comes to normal with computers, and especially in enterprises, there are "global" norms and "environmental" norms.  The globals are things like the 32-bit spawn from the 64-bit parent IE, the SCODEF references for child tabs (which includes the home page, by the way), and Outlook links spawning instances of IE to reach the websites.  Environmentals aren't out to save the computer, but are things like tracking software sitting in between services.exe and svchost.exe.  If you know what those are for your world, you'll be much better off when it comes to finding evil, and separating the signal from the noise.

Happy Hunting!