73

Yesterday, Twitter anounced that they recently identified a bug that stored passwords unmasked in an internal log. Recently, Github also had a similar bug. In both cases, they claim that nobody had access to these files.

Twitter:

We have fixed the bug, and our investigation shows no indication of breach or misuse by anyone.

Again, although we have no reason to believe password information ever left Twitter’s systems or was misused by anyone, there are a few steps you can take to help us keep your account safe:

GitHub:

The company says that the plaintext passwords have only been exposed to a small number of GitHub employees with access to those logs. No other GitHub users have seen users' plaintext passwords, the company said.

GitHub said it discovered its error during a routine audit and made it clear its servers weren't hacked.

To note, GitHub has not been hacked or compromised in any way.

How can Twitter and GitHub be sure that they have not been hacked or compromised in any way? Would someone who is compromising a server and read/copy a file always leave traces?

  • 63
    Twitter's statement is actually "shows no indication of breach" which could mean that if someone was there, there was no sign of it (this could be concluded from a bunch of different log sources that look at connections to and from said machine, which users had access to the machine in cause, etc). – sir_k May 4 '18 at 8:18
  • 42
    They did not state they are sure they haven't been hacked. They stated that they have no evidence of being hacked, but the absence of evidence is not proof of not being hacked - which they very wisely didn't claim it to be. – Adwaenyth May 4 '18 at 10:03
  • 4
    A recent article on this issue: "It’s Impossible to Prove Your Laptop Hasn’t Been Hacked." – André Paramés May 4 '18 at 12:40
  • 8
    "Hacked" isn't even the major category of risk here. Certain Twitter employees with legitimate access could steal these without any unauthorized penetration at all. In fact, that's almost certainly how this was discovered in the first place. – Beanluc May 4 '18 at 17:33
  • 3
    log file contents may travel unexpectedly. I once observed log file snippets exchanged between two supporters in the ticket system in a ticket I (external user) opened. The snippets were around the time of my problem, but contained many log entries concerning other users, including personal identification, cleartext passwords, cleartext security questions + answers, as well as relations between users (is assistant of). With the information, I could have dialed a phone number, ask for a specific name and tell them what their boss's favourite movie was ... So much for only exposed to a few – Hagen von Eitzen May 4 '18 at 21:19
118

They can't be sure. In fact, you can never be sure you haven't been hacked. But a thorough examination can make you conclude that it is more or less likely.

The Twitter statements only says that there is no indication of a hack. That doesn't exclude the possibility that they were hacked, and in urging their users to change their passwords they implicitly admits that.

As for GitHub, the wording is a bit more categorical. But I think forcing a password reset shows that they understand the risks involved.

  • 4
    "you can never be sure you haven't been hacked" - I cannot agree less. – Pedro Lobito May 5 '18 at 10:47
  • 34
    @PedroLobito Care to develop? – Anders May 5 '18 at 11:45
  • 14
    Merely air-gaped won't guarantee no hacks at all. Unless he means that for literally every component, for instance an independent power source instead of mains. Maybe even an air-gap wouldn't be sufficient, perhaps a vacuum.... But there is always user error, e.g. I'm aware for instance of air-gapped systems being phished per se with USB drives seeded in the parking lot. There is also the consideration that an air-gapped machine wouldn't be useful as a system users need to authenticate into and actually use.... – ttbek May 5 '18 at 14:00
  • 13
    You can be convinced you haven't been hacked, and eventually you will be wrong. – Theraot May 6 '18 at 16:43
  • 6
    Anyone is capable of building a system they can't hack. A corollary to that is: anyone can build a system where they can't detect that a hack has occurred. They are doing the best they can to examine their own environments for evidence of hacking. That they found no such evidence only proves that: they could find no evidence. It's up to you to determine whether or not their statement satisfies your own requirements for security. – Christopher Schultz May 6 '18 at 20:32
20

One more thing to note is that in both cases, the leak was in a purely internal logging system. There is no indication that 3rd party users ever had access to this system. Internal logging systems are rarely exposed externally, and only consulted internally when a system needs troubleshooting. That's also probably the reason why this bug went unnoticed for months: singular log entries somewhere in what's probably a gigantic amount of other statements usually don't get noticed unless they happen to be right next to or in the middle of statements that are needed to debug other entries.

Twitter also only recently found out about the bug themselves, which means it's unlikely that people from outside the company were aware of this bug before Twitter was, let alone figured out and executed an attack to retrieve them.

  • 6
    'Rarely exposed externally' - yeah, as long as you remember to secure your S3 storage... – djsmiley2k May 4 '18 at 14:07
  • 11
    It's not even about third parties. Employees of the first party can abuse this kind of thing just as readily. – Beanluc May 4 '18 at 17:31
  • Really, though, @djsmiley2k, how many major credit-reporting agencies actually store hundreds of millions of credit files on unsecured S3 buckets? – chrylis May 5 '18 at 21:02
  • 2
    @chrylis Apparently, most of them. – Ian Kemp May 7 '18 at 8:25
10

It's hard to prove a negative.

So how do you prove a positive? In this case: how do you prove an attack from the outside? Typically there are several systems in place to monitor different forms of attacks, breaches or access. These can be firewalls, intrusion detection systems, SIEMs and a variety of monitoring and logging systems. In today's networks each component either has some form of monitoring or is allowing monitoring through third party tools like Check_MK.

So each step of the way - from the border of the corporate network to the machine that held the valuable information itself - is in some shape or form monitored. These logs are, depending on the network and corporate policies, regularly analyzed. The analyzing systems can distinguish between expected and unexpected traffic or behaviour. Un/Expected behaviour is for instance file access.

Internal log files are typically considered confidential data, so file access is probably monitored as well. If someone that is not part of a certain user group tries to copy/access an internal log file, that would've probably been logged as unexpected or even forbidden behaviour. If a possible adversary was able to impersonate someone with the rights to access this file, it would've been logged as well, but as expected behaviour.

In theory it is possible that an attacker is able to overcome all security controls, exploit 0day vulnerabilities, leave no trace in every log on every component, the IDS, the SIEM and so on, copy the internal log file and smuggle it outside, but it is very unlikely.

My guess is, that after the log file was discovered, all these logs were thoroughly analyzed to try to prove if there was an attack from the outside. The analysts did not find any suspicious data and therefore concluded that with almost absolute certainty there was no attack from the outside. And this actually what you see in Twitter's press release (see Florin Coada's comment). Again, my guess: GitHub's press release had a more strict language to stop speculations if there was a hack. (Didn't really work out. ;)

Of course it's also possible that Twitter and GitHub have no such security controls in place, but I really hope not.

5

I assume they have access logs for just about anything happening on their servers, they can check whether someone accessed the file, when that was, what alias they logged in with, their source IP address, etc. If they can prove to themselves that all access was by legitimate employees they can confidently say they have not been hacked. Note that this does not actually guarantee them not being hacked, just that all evidence points in that direction.

1

The answer is very simple. Nowhere do they state they are sure. In fact, they implicitly tell you that there actually could have been a breach in two ways:

  1. They say that there is "no reason to believe there were" or "no evidence of" a breach. Lack of evidence could simply mean that the intruder(s) covered their tracks.
  2. They implore you to change your password, just to be on the safe side. This implies they cannot guarantee that no breach has occurred.
0

My interpretation of, especially of the more bold GitHub statement, is that they want to say that the fact that the passwords were put in the logfile is not the result of a hacking attack, but the result of a developer introducing debugging code by accident into production. This is relevant since an attacker might have introduced this functionality in order to collect passwords for grabbing them later.

There is no guarantee that they never have been hacked and won't ever be, but this incident is independent from hacking attempts.

0

Big companies like this should have lots of servers and therefore all outside access is routed through a bastion host (outside meaning not from within the actual server cage/room). The bastion might have say logging set up in a way since it relays all commands from the outside network into the operational servers. The commands if logged could easily be used to tell if someone opened a file with vim for example and that would solve the question of if a user had been hacked. SSH access is also logged so a known "good" IP/s can be filtered out for a user and any suspicious or odd entries could be examined by IT and if an entry has not been able to be explained then that would constitute a breach. Otherwise the server would be "safe" and there would not be as much of a need to worry regarding this matter.

0

What they are actually saying is that they are 100% sure that there was indeed a security breach. Accidentally. Probably. And by themselves. The rest is gloss.

They may view the individual differently, as an employee, but I don't. A good hacker works from the inside and gains trust. NSA? FSB?

They should never have plain text access to our passwords. That is not how password access works. The implications are that it is now up to you to change that password everywhere you ever use it. Assume the password is now known by everyone.

  • 4
    If having one password compromised means you have to go change it in a lot of places, that's your own fault. You shouldn't be doing that in the first place. – barbecue May 6 '18 at 6:11
  • @barbecue Also correct. This advise is for the mainstream. Everywhere can mean 0 to many places. – Sentinel May 6 '18 at 19:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.