Keep Track of File Acquisition with ProcMon

This post was inspired by Lenny Zeltser's recent blogs about using NirSoft Utilities in malware analysis.

Have I said that guy rocks? If not, let it be said. He rocks.

In having to do acquisition of loose native files across clients' LANs and WANs, you come across some difficulties, and slow networks. And large files, like 18GB NSFs that you find out you're pulling across the WAN from Arizona to Michigan. What? Yeah, no one realized that a guy in Michigan had his NSF in Arizona. No wonder he was complaining about email being slow... :) Anyway, aside from that, I've come across lots of situations where a file copy seems to be hung, but you don't want to kill it and try to start over in case it's actually still doing something. The things you want to know in those scenarios are:
* Has the Source already been copied to the Destination
* Has the Source already been hashed
* Has the Destination already been hashed
* Is the process still running/active
* How much longer is it going to take

Even if you could safely preview your destination without risking changing timestamps or otherwise messing something up, it would still show the full file size, regardless of how much has been written, so that's no good. And for the hashing bit, how could you really tell?

Some applications will provide status/progress updates. For instance, Pinpoint Labs' Safecopy will show progress, and even provide an indication of where you are in the process, but it can't tell you if it's still working or not. PixeLabs' XXCopy can provide a progress bar (and used with Jesse Kornblum's md5deep for hashing, which can also provide a progress report), Dan Mares' upcopy (my personal favorite) provides periodic updates, but none of these (or robocopy, RichCopy, etc) are able to tell you what you're actually dealing with. Enter SysInternals' Process Monitor, aka procmon.

You have to know how your tool operates, but if you do, procmon can be extremely helpful for discerning if you're still good to go, where you are in the process, and how much longer you can anticipate it taking (you'll have to do your own math). For my example I'm using upcopy, and copying several 10GB files.

upcopy command

Upcopy works by reading and writing the source to the destination. It then hashes the source, followed by the destination. At that point, it's done. Most apps work in pretty much the same way, but you have to know yours in order to make the best use of procmon.

Naturally, there will be a lot of activity in procmon, so we have to find our executable and set a filter to show only that. Click the binocular icon or CTRL-F to open the search feature. You can also just scroll down to find the exe you're looking for, as the search feature will give you every instance of it, which could give you results from antivirus scanning it, HIPS checking it, parent process, etc. Might take longer than what it's worth that way! If you're not sure the actual EXE name, find it first in SysInternals' Process Explorer.

Process Explorer

procmon, "find"

Once you locate your executable, right-click and select "include..."

procmon, "include exe"

By default, procmon show the most recent activity at the bottom, so you'll need to scroll down. I don't tend to change the configuration, sorting, etc once it's running, as it does use resources. Be aware and judicious in your choices.

The first set of screenshots here show the copy process, from source (F:) to destination (V:) (yes, I'm actually going local to network, the opposite of what we'd typically be doing in a collection; I needed to back up some files to my NAS anyway). You can see the "Offset" in the far right column increasing; this correlates to the file-size, and is one piece in checking progress.

copy progress

more copy progress

Once the copy process is complete, you'll see the activity in the next screenshot. Note the "ReadFile," "END OF FILE" and the "Offset" showing the total file size in bytes (yep it's a 10GB file), followed by the file being closed on source and destination. Then the hashing of the source will commence.

copy complete

You can follow the progress through hashing the same way. All the entries will show "ReadFile" action on the source, and you can keep tabs via the "Offset" value increasing. Note that by doing some math here (change in size over time passed), you can determine an approximate timeframe for completion of that portion.

hashing source

more hashing source

and yet more hashing source

Once it's finished hashing the source, you'll see the following. Just as before, note the "ReadFile," "END OF FILE" and the "Offset" showing the total file size, followed by the file being Closed. Then the process repeats for the destination.

source hash complete

Here's the destination hashing in progress; same as for the source, only the location is changed.

hashing destination

And, just as before, when the destination is finished hashing, you'll see it clearly in the activity in procmon. There's one other item to note here, though, and that is the logfile activity for the hash data.

destination hash complete

Lather, rinse, repeat, and that's about it. Do be aware that running procmon can consume system resources you might otherwise need (it lives off RAM). That's one of the reasons I like upcopy; it's no resource hog at all. And it's freakin' fast. I know Dan Mares thinks people don't use CLI anymore, but I beg to differ!

No great revelation of forensic truth here, just a little something to keep you in the know on what's happening with a file copy. Could also be applied to other processes where data is being written out and taking a long time, or when you need to more accurately calculate time estimates for the same (for some reason those application/system progress reports don't seem to be very accurate). I've found it useful; I hope others do as well.

Happy forensicating!

Would You Bet Your Life On It? Or Your Company?

It has been said that Information Security is Risk Management, and I agree with that. For any given situation, you have to identify vulnerabilities, threats (ie, "risk"), determine ways to mitigate these, and assign some value to that final level of risk. If that value is gauged to be acceptable to the organization (even if it's your family) then you move forward. But this isn't (or shouldn't be) limited just to Information Security groups - as I mentioned above, the same principle applies at home, on the road, and should also be in the minds of those not actively engaged in InfoSec positions. We live in a time and place in which threats abound, and Information Security is also not about saying "No" to everything; it's about figuring out how to say "Yes" where it's appropriate, and figuring out ways to reduce the risk (this also applies at home, on the road, with our kids, etc).

To keep this from being a totally pensive piece, I'm going to bring it back into the context of the work we do daily. As many of you are aware, a few months ago I experienced an abrupt change in job status, while working in digital forensics consulting. I'm still in a bit of a limbo situation (no, not dancing), but am working a contract gig doing information security. While there are business types that are defined as more at risk of cyber attacks due to industry, I think it should be obvious to everyone by this point that we're ALL under attack. I hear people say things like, "Well, we've never been breached, why do you think it would happen to us?" To that I respond, "You've never been breached? How do you know? Can you prove it?" I personally refer back to Dmitri Alperovitch's statement when talking about Shady Rat that in general he divides companies up as those that know they've been breached, and those that don't yet know.

So what's my point? Well, I'm getting there, albeit a little slowly. My point is that I think people today should have a general awareness of security risks, and that this should occur organically (ie, without having to be told). Even granted that mainstream media doesn't talk about APT, and only mentions the smallest percentages of places that are breached and lose integral control of their data, the info that does get out there should be sufficient. And yet, time and again, people buy cardboard iPads and MacBooks from criminals in gas station parking lots, fall prey to Nigerian email scams, and even fake IRS emails to install malware. But, even if common folk aren't hip to the threat, those in the IT industry will be, right? After all, they've all had to clean up after someone, they follow "geek" news not just mainstream, so they at least will get the fact that there are very real threats out there. Sadly, no.

I was at a presentation recently where a guy who's been working in InfoSec for 20 years told a story about his wife opening one of those IRS emails and following the link. She even put in her social security number when prompted. Then she complained of her computer acting strangely, and told him what happened. He "cleaned" the system by running a scan with an off-the-shelf antivirus/antimalware product, and went on, embarrassed that his wife had fallen prey to a scam. His opinion was that the situation was remediated. Really? You ran an AV scan and that's it? Did you analyze RAM, check network traffic, credit report activity, or do any investigation at all? Nope, just ran an AV scan and called it a day. Wow.

And recently at work we had an internal server that allows certain users to perform certain tasks, return odd results for one user. It was on a Monday morning, and results for that one user all appeared to be in Chinese. Do what? Yep, and just for that one user. We approached the admin about the situation, and as it turns out, on Thursday afternoon of the prior week, the admin for that server had installed some new patch rollups. Patch rollups, not fruit rollups. He felt it was probably related to the patches, as opposed to a compromise. Ok, sounds reasonable, but we still needed to play it safe. We pulled volatile data from the machine and started going through that while the admin investigated the patch scenario. We were quickly informed that the patches were to blame; the admin uninstalled and reinstalled (along with a few more), and said everything was good to go (yes, I realize evidence could've just been stomped on). And indeed, it appeared to be fine, and the explanation made sense. But we asked some followup questions nonetheless, and were greeted with the following response (not an exact quote): "I understand you think you're doing your job, but it was the patches, and it's been fixed. I have a lot of things to do, and don't have time to continue wasting on something that's been resolved." Wow, really? Our boss got involved, and there were some additional conversations...

My question when we received that response was, "Sure, it looks like that's what happened, but can you prove it 100%? Would you bet your life on it? Would you bet the company on it?" Because in essence that's what you're doing by turning and walking the other way, and if you're not willing to bet it all, it's probably the wrong answer. No root cause analysis, and with all the companies falling victim to basic compromises, and allowed to bleed data for who knows how long, and you're willing - as an intelligent IT admin - to say that a system which was serving up Chinese characters is good to go, because of a patch? That seems like a bit of a blind risk. It would have taken so little time to go through our followup questions and answer them, and that would have helped shed great light on the situation, to give us all more peace of mind. I guess I shouldn't be surprised, though. And even if I become so jaded that I'm no longer surprised, I think I still get to be disappointed. ;)

Do I have a solution? I wish I did, but I don't. I understand that education is paramount, but I think it takes more than that. I think it takes an awareness and understanding that there is a clear and present danger (er, threat), and a desire to be part of the solution, rather than part of the problem. And that's what I think is lacking - the desire. The IT guy should already have the knowledge, and the InfoSec guy should have the knowledge. And those are just two examples; I could also talk about the IR guy who has no problem - doesn't even give it a second thought - connecting his laptop up to public wireless. Do we just get complacent and lazy as humans? Or is it that some of us aren't driven and determined to make a difference, and are just trying to get by until it's time to go?

Well, I think that's about all I have. I do want to take a moment to say that there are a lot of folks out there who are driven and determined to make a difference. Just take a look at the blogs I read, for a very small selection. I don't really do the #FF thing on twitter, but I'll give a shout out to Lenny Zeltser as I find his blog extremely practical and helpful. I don't think a post goes by that I don't get something very useful out of it. Thanks for sharing!

For those in the US, have a wonderful Labor Day weekend! For everyone else, get back to work! :) And since our attackers don't honor holiday weekends, be alert; we obviously need more good lerts! :D