150

In the UK, the company TalkTalk was recently hacked.

It was later discovered, after 'investigation' that the hack was not as serious as it could have been (and less than expected).

I'm wondering: How do organizations (not necessarily TalkTalk -- that's just what prompted me to ask) check what has been hacked? I'm sure there are many ways; but what are the 'main' ones?

  • 1
    In this case - probably by asking the hacker: wired.co.uk/news/archive/2015-10/26/talktalk-arrest – Sobrique Oct 28 '15 at 10:01
  • 4
    Besides all the great answers, and only as a side-point comment: The public most often gets a result maybe resulting from forensics, in which case the answers apply. Consider also, that any result like "oh, it was really not thaaat bad", can only be a bloody lie of the companies. So with respect to determine "what has been hacked", a substantial part of what the public sees can also be have traces or PR and marketing and not only forensics. Even though maybe you choose to disregard this when posing the question, the picture would be incomplete without this mentioned – humanityANDpeace Oct 28 '15 at 10:06
  • @humanity I guess that's possible, I didn't really think of it like that.. :/ – ᔕᖺᘎᕊ Oct 28 '15 at 10:19
  • 4
    IMHO they haven't actually discovered specifically what was hacked they have simply guessed based on what was leaked and said that as a PR statement to try to lessen the perceived damage. Given the poor level of security in the first place I doubt they have secured logging that can be trusted. – JamesRyan Oct 29 '15 at 14:49
  • @JamesRyan I also thought the same, as the statement was very quick and seemed odd, but honestly, I don't care about TalkTalk :P -- it was just an interesting topic that made me curious. I just used TalkTalk as an example. – ᔕᖺᘎᕊ Oct 29 '15 at 15:06
175

In a word: Forensics.

Computer forensics is the art of examining a system and determining what happened upon it previously. The examination of file and memory artifacts, especially file timelines, can paint a very clear picture of what the attacker did, when they did it, and what they took.

Just as an example - given a memory dump of a Windows system, it is possible to extract not only the command lines typed by an attacker, but also the output that they saw as a result of running those commands. Pretty useful in determining impact, eh?

Depending on the freshness of the compromise, it's possible to tell quite a lot about what happened.


@AleksandrDubinsky suggested that it would be useful to outline the various Computer Forensic fields and techniques, which I'm happy to do for you. They include, but are not limited to, the following (I'm going to use my rough terms; they aren't official or comprehensive):

Log/Monitor Forensics: The use of 'external' data such as centralized logs, firewall logs, packet captures, and IDS hits straddles the line between "Detection" and "Forensics". Some of the data, like logs (even centralized) cannot be trusted to be complete or even truthful, as attackers can filter it or inject it once they have control of the system. Off-the-system packet logging tools like the firewall, IDS, or packet recorders (such as (formerly)NetWitness) are unlikely to be tampered with, but contain only a limited amount information; usually just a record of IP conversations and sometimes signatures (such as HTTP URLs) associated with malicious activity.

Unless an unencrypted network connection was used in the compromise, these tools are rarely able to detail the activity during a compromise, and so (going back to the original question) don't "check what has been hacked." On the other hand, if an unencrypted connection (ftp) was used to exfiltrate data out, and full packets were recorded, then it's possible to know exactly what data the attacker ran away with.

Live Forensics: More properly part of Incident Response, so-called "Live" forensics involves logging into the system and looking around. Investigators may enumerate processes, look in applications (e.g. browser history), and explore the file system looking for indications of what happened. Again, this is usually designed to verify a compromise happened, and not to determine the extent of a compromise, since a compromised system is capable of hiding files, processes, network connections, really anything it wants to. The plus side is that it allows access to memory-resident things (processes, open network connections) that aren't available once you shut the system down to image the disk (but, again, the system may be lying, if it's compromised!)

Filesystem Forensics: Once a copy of the disk has been made, it can be mounted on a clean system and explored, removing any chance of the the compromised operating system "lying" about what files are in place. Tools exist to build timelines using the variety of timestamps and other metadata available, incorporating file data, registry data (on Windows), even application data (e.g. browser history). It's not just file write times that are used; file read times can indicate which files have been viewed and also which programs have been executed (and when). Suspicious files can be run through clean antivirus checkers, signatures made and submitted, executables loaded into debuggers and explored, "strings" used to search for keywords. On Unix, "history" files - if not turned off by the attacker - may detail the commands the attacker entered. On Windows, the Shadow Copy Service may provide snapshots of past activity that has since been cleaned up.

This is the step most commonly used to determine the scope and extent of a compromise, and to determine what data may or may not have been accessed and/or exfiltrated. This is the best answer to how we "check what has been hacked."

Disk Forensics: Deleted files disappear from the filesystem, but not from the disk. Also, clever attackers may hide their files in the "slack space" of existing files. These things can only be found by examining the raw disk bytes, and tools like The Sleuth Kit exist to rip that raw data apart and determine what it means about the past. If there was a .bash_history file, and the attacker deleted it as his last step before logging out, then that file may still exist as disk blocks, and be recoverable. Likewise, attack tools that were downloaded and temporary data files that were exfiltrated can help determine the extent of the compromise.

Memory Forensics: If the investigator can get there soon enough, and get a snapshot of the memory of the system involved, then they can thoroughly plumb the depths of the compromise. An attacker must compromise the kernel to hide from programs, but there's no way to hide what's been done if a true image of kernel memory can be made. And just as the disk blocks contain data that has since been removed from the filesystem, the memory dump contains data that was in memory in the past (such as command line history and output!) that has not yet been overwritten... and most data is not overwritten quickly.

Want more? There are training programs and certifications to learn Forensics; probably the most common publicly available training is from SANS. I hold their GCFA ("Forensic Analyst") certification. You can also review the Challenges from the Honeynet Project, in which parties compete to unravel the cases given disk images, memory dumps, and malware samples from actual compromises - they submit their reports of what they found, so you can see the sorts of tools used and findings made in this field.

Anti-Forensics is a hot topic in the comments - basically, "can't the attacker just hide their traces?" There are ways to chip away at it - see this list of techniques - but it's far from perfect and not what I'd call reliable. When you consider that "the 'God' of cyberespionage" went so far as to occupy the firmware of the hard drive to maintain access to a system without leaving a trace on the system itself - and still got detected - it seems clear that, in the end, it's easier to detect evidence than it is to erase it.

  • 2
    Some companies may also have retroactive log analysis as a tool, where data on intrusions is collected, but not acted upon. This is worryingly common where the security of the business is not considered as worthy of full time staff. In this case, logs may contain all the details about the data accessed, but no-one is looking at them until a breach is found through other means, whether media reports, or customers complaining that their accounts have been accessed. – Matthew Oct 27 '15 at 11:50
  • 8
    Wouldn't a concerned attacker clean all that information prior leaving the system? In the end memory is limited, the attacker can either be able to access that part of the memory to wipe it or he can flood that memory to replace old contents with new ones. Couldn't he? – YoMismo Oct 27 '15 at 14:18
  • 29
    @YoMismo, the short version is, "It's not that easy." While many attackers will try to clear their tracks from disk, even that's not easy - that's what timeline analysis is all about, inferring activity from metadata. Very few go further and try to limit their memory footprint, and it's impossible to do totally - at the very least, you leave a footprint of your footprint clearing program! “All data leaves a trail. The search for data leaves a trail. The erasure of data leaves a trail. The absence of data, under the right circumstances, can leave the clearest trail of all.” - C.S. Friedman – gowenfawr Oct 27 '15 at 14:37
  • 1
    Thanks for adding the details. I agree with Alex that before the addition of the detail the answer just points to an equally undefined word of "forensics". For an answer to be useful you have to talk at a level familiar to the audience. If you know what computer forensics are, you already know the answer to the question. – Steve Sether Oct 27 '15 at 19:06
  • 3
    @YoMismo, ...once upon a time, when designing a particularly sensitive system, I did logging over a serial line with the pins for read communication cut -- literally, a write-only interface, with the system on the other end accessible only by someone physically entering a secured area. There's quite a bit one can do when the situation warrants. – Charles Duffy Oct 27 '15 at 23:04
20

Working within the technology sector, how I would do it would be as follows:

  1. Find the hack, the hole, the weak spot.

  2. Check the syslog, dmesg, access.log and error.log to confirm the thesis. (when hacking in to a network you normally have the break the system in the case of TalkTalk it was a MySQL injection, doing this would have left errors in the mysql.err log as to break in you have to break it.)

  3. Replicate the hack, replicating the hack will allow you to see how deep the hack can go and allow you to estimate the damages.

If you find the hole, you can replicate the attack, replicate the attack and you will see what the attacker saw.

  • 4
    This answer is a good addition to the other answers, since it addresses that a 'hack' does not in general give full access to a system, but is more often than not a leak in a specific part of the system. – Sanchises Oct 27 '15 at 12:42
  • 6
    Why didn't you used a numbered list? (you can replace , with . to do this) – A.L Oct 27 '15 at 13:01
  • 6
    because i didn't want to. – TheHidden Oct 27 '15 at 13:18
  • 4
    I don't think this is accurate. Firstly by 'MySQL injection', I assume you just mean 'SQL Injection', it's not a vulnerability that's limited to MySQL. Secondly, I don't think there would necessarily be any trace in the database's error log, if my injection attack is to change: ...where userId = '123' into ...where userId = '' or 1=1 this is perfectly valid SQL and the database would not identify it as an error. – DaveyDaveDave Oct 27 '15 at 14:02
  • 1
    @silverpenguin: reasonable points, but all I was trying to show is that your general assertion of "to break in you have to break it", is not necessarily true. – DaveyDaveDave Oct 27 '15 at 14:36
7

In addition to the excellent answer on Forensics (and comments) some organisations use intrusion detection systems such as 'Snort' (network intrusion detection) and OSSEC (host based detection system) as well as many others by vendors such as Cisco etc. which record various aspects of unusual or unauthorised behaviour.

For example, host based intrusion detection systems often log file changes on servers so a forensic investigation can show which files have been changed within a given period and a forensic investigator can use this to piece together activities of some users or attackers.

Attackers will also leave traces in logs all over the infrastructure from firewall logs, to server logs, service logs (web, smtp etc.) and these can provide vital clues. However, it is possible for attackers to change these logs unless they have been designed to be secure and unalterable e.g. by centralising logs on a write-only medium.

Once an attacker gains privileged access the gloves are off and logs can be tampered with or deleted.

In the end it will rest on readiness - whether the organisation has prepared for intrusion and the forensic exercise that will follow. This should be part of every company's security planning.

  • 2
    It will also depend on whether the attacker finds/knows about every log. – Dan Henderson Oct 27 '15 at 15:00
  • @DanHenderson but how do you know the difference between an unfound log and a planted false log? – JamesRyan Nov 3 '15 at 10:56
  • Because you need a way to ensure that logs can't be tampered with (or at least require collusion in order to do so). The best way to achieve this is to either write logs to write-once medium (in *nix systems you can mount a drive as 'append only') or write to a secure, centralised log server. – David Scholefield Nov 3 '15 at 12:12
  • @JamesRyan a found, falsified log will agree with the other found, falsified logs. An unfound log will contradict the falsified logs. – Dan Henderson Nov 3 '15 at 17:06
5

I haven't seen this mentioned yet and it is likley a major resource for such an investigation, especially one revolving around data access:

RDBMS auditing can be a major resource for determining exactly what data has been accessed, when, and by whom. This of course depends on how, or if, it has been configured.

For example, SQL Server Audit allows retrieval of user IDs, time stamps, and the actual queries submitted to the server thus allowing complete re-construction of the data set stolen.

The audit systems should have redundancies built in to prevent tampering with the audit trail. Some of this may be stored in the RDBMS, and some may be written to external sources. SQL Server can log audits to the Windows Security log.

4

gowenfawr mentioned system forensics but identifying the compromised asset is almost always done via logs. Logs /should/ always indicate the compromised asset if you search hard enough. Whether you have an IDS that detects external logins/bruteforce attempts/SQL injection attempts or if you manually search logs for what I just mentioned, there are several ways to bring a breach to your attention.

Once the asset has been identified, you can begin forensic examination of the asset. Mostly for additional information that logs may not report, especially if it looks like a file has been dropped to maintain access for the attacker. Memory scraping and analyzing what the dropped file does, step-by-step, will help the security team/IR team to gain an understanding of what's going on, how the compromise initially occurred, the vulnerability utilized and potentially the source of the attack (the real source).

3

They can check several things that I can mention randomly:

  • Syslog
  • Firewall's logs
  • Logs of IDS and router(s)
  • File system
  • Events
  • Scheduled tasks
1

If someone broke into your house using a stolen key, and did not steal anything, how would you know? Often, you would. Think about how the ways in which you'd detect the occurrence. It could be as silly as a letter open on your kitchen worktop that you don't remember getting around to reading. There is basically no distinction here. It helps if you find scratch marks on your lock from the first ten entry attempts before the burglar finally determined which key was actually yours (read: security logs).

  • 1
    what are you talking about? – user45139 Oct 28 '15 at 10:37
  • 5
    @Begueradj: It's a pretty clear analogy. With which part of it are you having trouble? – Lightness Races with Monica Oct 28 '15 at 10:42
  • 2
    @Begueradj: I understood it perfectly, thank you very much. Forensics may be modelled to the layperson as analogous the way a homeowner examines his or her property for signs of damage/loss that may not be outwardly evident without such activity. The OP was ostensibly only considering the case where the sofa and TV are missing and therefore could not think how more subtle forms of theft/infiltration could be detected. That led to this question being posted and, ultimately, this answer. Let me know if you have any more questions. – Lightness Races with Monica Oct 28 '15 at 11:00
  • 3
    @schroeder: No, it is an analogy. – Lightness Races with Monica Oct 28 '15 at 16:12
  • 2
    @schroeder I've never read ellipsis used in any sense even remotely like this, and my search doesn't find any related definitions. It would've helped to include a citation for those of us whose English is apparently lacking. – underscore_d Oct 31 '15 at 10:48
1

In terms of what has been taken, rather than how, bigger enterprises and agencies utilise deep packet inspection to record what has been transferred in/out of their network.

Devices record all traffic and can utilise IDS for alerting. After an incident occurs they can analyse the magnitude of the attack.

Some common industry standard tools for this are:

1

In the case of Talk Talk, we know that the stolen data is being used to target people.

  • We know these people are being phoned up by someone claiming to be from Talk Talk and then tricked into providing their bank details.
  • We know that bank details stop being useful after a short time if it is know then have been hacked.
  • We therefore know that hacked bank details would be used QUICKLY if taken from TalkTalk.
  • Therefore it is reasonable to assume that the bank details were not in the data that have been taken.

There is a LOT of monitoring going to by the UK banks etc to see what the impact is, this monitoring will pick up enough cases, along with reported cases by bank customers, to give a good ideal of the impact by now.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.