How can Twitter and GitHub be sure that they haven't been hacked?

Question

Yesterday, Twitter anounced that they recently identified a bug that stored passwords unmasked in an internal log. Recently, Github also had a similar bug. In both cases, they claim that nobody had access to these files.

Twitter:

We have fixed the bug, and our investigation shows no indication of breach or misuse by anyone.

Again, although we have no reason to believe password information ever left Twitter’s systems or was misused by anyone, there are a few steps you can take to help us keep your account safe:

GitHub:

The company says that the plaintext passwords have only been exposed to a small number of GitHub employees with access to those logs. No other GitHub users have seen users' plaintext passwords, the company said.

GitHub said it discovered its error during a routine audit and made it clear its servers weren't hacked.

To note, GitHub has not been hacked or compromised in any way.

How can Twitter and GitHub be sure that they have not been hacked or compromised in any way? Would someone who is compromising a server and read/copy a file always leave traces?

Answer 1

They can't be sure. In fact, you can never be sure you haven't been hacked. But a thorough examination can make you conclude that it is more or less likely.

The Twitter statements only says that there is no indication of a hack. That doesn't exclude the possibility that they were hacked, and in urging their users to change their passwords they implicitly admits that.

As for GitHub, the wording is a bit more categorical. But I think forcing a password reset shows that they understand the risks involved.

Answer 2

One more thing to note is that in both cases, the leak was in a purely internal logging system. There is no indication that 3rd party users ever had access to this system. Internal logging systems are rarely exposed externally, and only consulted internally when a system needs troubleshooting. That's also probably the reason why this bug went unnoticed for months: singular log entries somewhere in what's probably a gigantic amount of other statements usually don't get noticed unless they happen to be right next to or in the middle of statements that are needed to debug other entries.

Twitter also only recently found out about the bug themselves, which means it's unlikely that people from outside the company were aware of this bug before Twitter was, let alone figured out and executed an attack to retrieve them.

Answer 3

It's hard to prove a negative.

So how do you prove a positive? In this case: how do you prove an attack from the outside? Typically there are several systems in place to monitor different forms of attacks, breaches or access. These can be firewalls, intrusion detection systems, SIEMs and a variety of monitoring and logging systems. In today's networks each component either has some form of monitoring or is allowing monitoring through third party tools like Check_MK.

So each step of the way - from the border of the corporate network to the machine that held the valuable information itself - is in some shape or form monitored. These logs are, depending on the network and corporate policies, regularly analyzed. The analyzing systems can distinguish between expected and unexpected traffic or behaviour. Un/Expected behaviour is for instance file access.

Internal log files are typically considered confidential data, so file access is probably monitored as well. If someone that is not part of a certain user group tries to copy/access an internal log file, that would've probably been logged as unexpected or even forbidden behaviour. If a possible adversary was able to impersonate someone with the rights to access this file, it would've been logged as well, but as expected behaviour.

In theory it is possible that an attacker is able to overcome all security controls, exploit 0day vulnerabilities, leave no trace in every log on every component, the IDS, the SIEM and so on, copy the internal log file and smuggle it outside, but it is very unlikely.

My guess is, that after the log file was discovered, all these logs were thoroughly analyzed to try to prove if there was an attack from the outside. The analysts did not find any suspicious data and therefore concluded that with almost absolute certainty there was no attack from the outside. And this actually what you see in Twitter's press release (see Florin Coada's comment). Again, my guess: GitHub's press release had a more strict language to stop speculations if there was a hack. (Didn't really work out. ;)

^{Of course it's also possible that Twitter and GitHub have no such security controls in place, but I really hope not.}

Answer 4

I assume they have access logs for just about anything happening on their servers, they can check whether someone accessed the file, when that was, what alias they logged in with, their source IP address, etc. If they can prove to themselves that all access was by legitimate employees they can confidently say they have not been hacked. Note that this does not actually guarantee them not being hacked, just that all evidence points in that direction.

Answer 5

The answer is very simple. Nowhere do they state they are sure. In fact, they implicitly tell you that there actually could have been a breach in two ways:

1. They say that there is "no reason to believe there were" or "no evidence of" a breach. Lack of evidence could simply mean that the intruder(s) covered their tracks.
2. They implore you to change your password, just to be on the safe side. This implies they cannot guarantee that no breach has occurred.

Answer 6

My interpretation of, especially of the more bold GitHub statement, is that they want to say that the fact that the passwords were put in the logfile is not the result of a hacking attack, but the result of a developer introducing debugging code by accident into production. This is relevant since an attacker might have introduced this functionality in order to collect passwords for grabbing them later.

There is no guarantee that they never have been hacked and won't ever be, but this incident is independent from hacking attempts.

Answer 7

Big companies like this should have lots of servers and therefore all outside access is routed through a bastion host (outside meaning not from within the actual server cage/room). The bastion might have say logging set up in a way since it relays all commands from the outside network into the operational servers. The commands if logged could easily be used to tell if someone opened a file with vim for example and that would solve the question of if a user had been hacked. SSH access is also logged so a known "good" IP/s can be filtered out for a user and any suspicious or odd entries could be examined by IT and if an entry has not been able to be explained then that would constitute a breach. Otherwise the server would be "safe" and there would not be as much of a need to worry regarding this matter.

Answer 8

What they are actually saying is that they are 100% sure that there was indeed a security breach. Accidentally. Probably. And by themselves. The rest is gloss.

They may view the individual differently, as an employee, but I don't. A good hacker works from the inside and gains trust. NSA? FSB?

They should never have plain text access to our passwords. That is not how password access works. The implications are that it is now up to you to change that password everywhere you ever use it. Assume the password is now known by everyone.