Recently, I was channeling my inner rock star, and thought I'd share a finding regarding "normal" occurrences. You're probably all familiar with LSASS.exe, the "Local Security Authentication Subsystem Server" process, and you might also know that it doesn't have any children. Poor thing; children are truly a gift (and a challenge, but that's a different topic). Anyhow, as noted in the SANS DFIR "Find Evil" poster, if LSASS spawns a child process, it bears looking into - and that's exactly what I was doing.
Given that I think it's important to be as proactive as possible with regard to incident response, I am always looking for ways to spot potential problems. Now, the SANS poster showcases things to watch for when doing memory analysis, but if you're parsing all executable activity in real time and storing that data in a way it can be queried at will (kind of like Sysinternals procmon on steroids), then why not apply the same principle and see what can be found? Yes, yes, I'm talking about CarbonBlack (now part of Bit9), which is (in my opinion) an awesome endpoint monitoring platform. However, while this post will make use of that technology, don't think of it as being about Cb, but rather about the hunt, what's found, and how that informs the bigger picture (and may change some of what's considered "normal"). Keep in mind there are other tools that can help accomplish the same goal, and as noted, memory analysis (with tools like Volatility, Rekall, and Mandiant Redline) is at the forefront - so don't get hung up on the tool; it's the investigator that makes the difference in the long run.
With that, I'd like to tell you a story about the hunt for spawn of LSASS, and how it started with a simple little query, as shown below; basically, any process for which LSASS.exe showed up as the parent...
Right away we see that there's one sweet child o' lsass, on 37 endpoints, with two different hashes showing up. Okay, so two different binaries or versions, then. Let's keep digging, starting with a listing of the search results.
A couple things stand out; namely that each process is associated with six (6) filemods (such as create/modify/delete a file) and two (2) netconns (could be browsing to a website, IP address, hostname), while related activities for registry (regmods) and other processes/binaries (modloads) are similar in count but different. If you're the type (and I am) that likes to review data offline to filter, sort, search, and so on, you can download a CSV that looks a little something like what's below. If you have oddball md5 values, abnormal paths, or process names that stand out, it's sometimes easier to focus in on (at least for me). With this, the "start" value is the time the current instance of the process ran, and "last_update" is the most recent time it actually did something as it applies to the query at hand.
I mentioned oddball md5 values, right? And we know we have two different ones at play here, and 37 total processes, so what's the breakdown? Funny you should ask (well, I asked, but it was kind of rhetorical anyway - just work with me...)
So apparently two out of the 37 are the "f17e" hashes, with the remainder being "bcb8." And yes, that can be identified using the GUI, but I like to see things with my own two eyes and plus this is an offline record in case I ever need it. No abnormal paths were noted.
Anyway, since there's consistency with filemods and netconns, that's a good place to start looking, but first, I'd like to know a little more - high level - about these processes. From a tool standpoint, Cb has a "preview" feature, and so to take a peek at the different binaries involved here (remember, two different hash values)...
First we had "Mr. Popularity," the "bcb8" version, followed up by the indomitable "f17e." In either case, the command line parameters are the same: "/efs /enroll /setkey." If you had not already, you're probably running out and searching teh intarwebs for this executable, EFS, and whatever else might give some insight, since it appears to be from Microsoft (R) and might be legit (but you never know, right?). If that's what you're doing, no worries, I was in the same boat. I even reached out to the SANS DFIR email list to see if anyone else had encountered this, since all know that "normal" means LSASS doesn't have kids. No children. Nada. Zip. Zilch. Carlos Marins pointed me to the following MS document (link is direct download) http://download.microsoft.com/download/9/8/2/98207DD4-7D2C-4EF6-9A9F-149C179D053E/CommercialOSSecFunReqsPublicV2Mar09.docx, which was very helpful in understanding some of the things I would subsequently find. Chris Pogue asked a few questions and reminded me about checking the hash with Bit9 File Advisor.
Speaking (er, well, writing) of hash, we'll go ahead and knock that out. VirusTotal and File Advisor both came up clean:
Alright, now back to the fun stuff - filemods and netconns for this sweet child o' lsass. Oh, first - this'll be quick, I promise - we can take a look at more details about the two binaries in question (as binaries, not as the named processes).
In addition to a few more details, this will show how many times that particular binary has been seen in action, without any time or other filtration (such as by parent process). In addition, using the aptly-labeled "Download" button, I can extract a copy of the raw binary for additional analysis, reverse-engineering, and so forth, offline. More on that later, as I said this would be quick. Now, back to the rest!
Here's a quick look at each of the processes for analysis:
You can see that the relationship between wininit.exe and lsass.exe is as expected. It's just that the latter spawns efsui.exe as a process (which of course we already knew by this time). We see the commandline parameters again, and the fact that it's signed by MS. What's new here is that the username (obfuscated) is the domain account of the actual end-user; it's not System or other non-human (actually, it's in the binary preview window as well, but I really just wanted to call it out here, rather than there). Also, pay attention to the "Export events to CSV" button in the top right (more on that later as well). And, more of the same from the other version of the process...
We were going to look into filemods and netconns if I recall correctly (and of course I do), so don't stop now...
You can't see the whole screen above, but just underneath the process map, there are some "facets" to speed up queries/drills/searches based on different criteria (of course, we're going to look at filemods and netconns - aren't you just tired of hearing about how we're "going" to do that?).
Clicking on filemods highlights some other areas within that category of activity, and also focuses our results on just those. Thus, we can see that the actions are broken down equally between creation and first write, and that only three directories were utilized, all within the user profile under AppData\Roaming (clicking on any of those would highlight only those pieces of information, thus narrowing the search further). Next up are the event details.
Here, in timeline fashion, we get to see file creation, first write (if there were any file deletes, we would see those too), and some details associated with each. The expanding arrow (as shown on the top entry) shows frequency information (singular in this instance because of the nature of the path), and the "Search" (in blue) will take us to those results (for instance if there were multiple systems instead of just one, we'd get to see what those all were). The "Search" box at the top right of the event list allows us to find any search string in the results (such as username, filename, part of the path, etc), if we had something we wanted to jump to quickly (or even just see if it was present). What's of interest here are the paths involved, which start to make sense in light of the MS document I mentioned earlier (same for regmods, but I'm not going to go into those for the sake of brevity).
Clicking on filemod in the facets again clears that drill, and we can switch over to netconns.
Basically, the two netconns were pointing to domain controllers, and as can be seen from the frequency information, those DCs are quite common from a connectivity perspective (as one would expect, being DCs). The interesting thing here are the ports involved, which makes it look like LDAP is involved.
Okay, we're getting down to the wrap-up stage (thanks for sticking with me this far, I know it's been a lengthy post/novella at this point). What it appears we have are known, signed, MS executables associated with the Encrypting File System (EFS) for transparent file encryption, reaching out as the logged-in user to domain controllers for authentication, using LDAP. But is that really truly what's going on? Do we know enough to say that at this point? Are there any additional checks we could do, ways to validate/verify this theory?
What about looking at other activity on the system for suspicious processes, network communications, or known malware? Any evidence to suggest that EFSUI.exe had been injected with other code after it started running? What about packet captures for these LDAP connections - are they really normal authentication for the process? Any corroborating logs from firewalls, the DCs, etc? Is EFS used in the environment, or is the user known to do so specifically? What about the binaries involved - does reverse engineering (RE) indicate any oddities or abnormalities?
Do you remember that there is the ability to extract a binary for offline analysis? So RE is definitely a possibility. Firewall and DC event logs should be reasonable to expect (although not necessarily a given). Packet captures from the time of the event (in the past) would require some dedicated NSM for streaming pcaps, but would be a really good way to help determine what's going on inside those netconns. And we can certainly dig into the more detailed process and binary analysis on the hosts in question. That "Export events to CSV" button I mentioned earlier? Gives output like this, with a CSV for each "type" of activity, and will also include child processes if available (there weren't any for efsui):
Note: The "Summary" text file gives some info about the process or binary being analyzed (name, hash, path, etc).
These spreadsheets provide a veritable plethora of information about the process or binary being analyzed. Don't know if you noticed, but there was a section on the analysis page, referencing "Alliance Feeds" - this provides info about matches to Virus Total, known bad domains, and other "intel" related to activity that might be full of evil. Rather than a specific process to search for, you could also start with a given endpoint/host, any of these threat feeds, or some custom query for known indicators (based on firewall, IPS, or other threat "intel" you have). Anyway, all that to say that if you wanted to correlate the activity from EFSUI.exe as a child process of LSASS.exe, to any other activity to help determine whether this benign or the most evil thing on the planet, there's a lot you can do. And again, not just with CarbonBlack - you can set up a packet capture for a period of time (host, router, switch, firewall, etc), perform memory analysis, targeted triage (such as popularized by Harlan Carvey, Corey Harrell, and Chris Pogue to name a few key folks), or even (gasp!) a full disk image with timeline and analysis galore!
Maybe the information here is enough to make a determination and speak authoritatively about what happened, and whether there are any unremediated or ongoing risks. However, it's really ultimately about those risks, and as investigators we may not be the decision-makers (in fact, most likely are not). We can inform the decision-makers of our findings and recommendations, but we also have to be honest and explain what options are available, what those options would provide (and at what cost), and what potential risks could be incurred (and the probability thereof) by not pursuing the aforementioned options. Want to know more on this subject? Come to my talk at the SANS DFIR Summit in June - To Silo, or Not to Silo: That is the Question. More info is available about the Summit here.
Thanks again for "listening" to my tale about the sweet child o' lsass, and remember ... you may know that lsass doesn't have any child processes, but if you don't verify or validate that, you might just reach the wrong conclusions, and that probably wouldn't be good in a "real life" scenario.
Happy Hunting!
No comments:
Post a Comment