2

I recently found that a big company use an outdated product in their file-storage servers (I got the hidden name and version). The bug is corrected in recent version, but they weren’t any cve or exploit until I create them. (their is still no cve and many distributions still sheep affected git packages in current branches)

They do set limits on ram usage on their http backends but not on their virtual storage servers (a single process can stall the server on swaps)
They do per account load balancing which means each storage server contains the entire data of http accounts (they have hundreds thousands accounts). I was unable to find how many they have, but considering their site is a top web site (in terms of worldwide traffic) I just guess this should be a lot.

I tried the issue and finally crashed one of their storage server during several hours with a single crafted request (the http server was sending 500 pages to the users trying to access the accounts stored on it)
I apologised and mailed them about the behaviour. I raised the concern of a larger scale attack that would shut down all file servers leaded by a team of 5~10 peoples without requiring botnets.

Hello again,

In these cases we feel that our compensating controls can make this significantly harder to exploit. You are probably correct that all of these things are possible and ideally would be fixed, but we consider this to be very low risk. I'm very impressed with the investigations you have done !

They really seems to treat this like classical attacks from botnets affecting bandwidth.

I learned to always update software that suffer from exploitable memory leaks. However I’m only a student, I have no experience at managing such a large website (~47 valid requests per minutes). But here are some points :

  • According to their status page, none of the classical dos attacks they faced impacted users (which isn’t the case here)
  • While their “controls” might prevent the whole site to be put offline, it definitely won’t prevent a large downtime of several of their storage servers (affecting thousands users).
  • The purpose of the site is to host software, it’s perfectly possible to target the most downloaded ones.
  • Since storage servers have 50Gb of ram I don’t think they have so much of them (I don’t know if large needs like this can require a >20Tb of ram)
  • It can only be used for dos. There is no longer possibility for things like remote code execution.
  • Once a storage server recovered, it’s possible to put it down again immediately since nothing prevent a crafted request to be made from a different machine with a different ip and it don’t require to log‑in (then the repository is unavailable most of the time for several days)
  • In the meantime, they do many things to prevent a push request from filling ram, like limit the maximum object size, or disallow deep nested tree objects. They also took actions on a prior problem allowing ram exhaustion from one of my report.

Something I can’t understand is they worry about non‑root remote code execution on a single of those server whereas the software that power them is public (they don’t worry about the fact a large part of their configuration is accessible from outside).

So are there situations where keeping exploitable memory leaks is Ok ? (only 1.2Gb of network data is required for stalling a 50Gb server)
Or more generally, should ram be limited on ɢɪᴛ servers ?

|improve this question

migrated from serverfault.com Nov 29 '15 at 15:03

This question came from our site for system and network administrators.

  • This is a second e-mail I posted here. I already insisted on the behaviour. – user2284570 Nov 25 '15 at 21:42
  • > They really seems to treat this like classical attacks from botnets affecting bandwith. Actually, there are many classical attacks from botnets which affect more then just network bandwidth. Some DOSes attempt to overload the CPU, and some even try to exploit memory leaks. Your description sort of sounds like the second example. – Stefan Lasiewski Nov 25 '15 at 22:43
  • @StefanLasiewski : In the current case, 1Mb of network data fill 900Mb of ram on the server. – user2284570 Nov 25 '15 at 22:44
  • If you truly feel that you've discovered a serious security problem & feel like you need to escalate the notifications, be sure to read up on guidelines pf "Responsible disclosure". These guidelines are intended to give the vendor time for internal review & testing before the CVE becomes public, and can protect other users of the product from attacks "in the wild". In return for responsible disclosure, you can get an acknowledgement in the CVE and improved reputation in the community. – Stefan Lasiewski Nov 25 '15 at 22:55
  • @StefanLasiewski : the official git project don’t have private mailing lists. The amplification came with ssh compression (otherwise, 255 bytes of network data fill 255 bytes on the server). – user2284570 Nov 25 '15 at 22:57
7

The answer to this comes down to a risk-management decision that the organization makes. There are best practices, but even the bestest of practices can be compromised by risk-management decisions. There are no absolutes, even patch broken stuff is not an absolute.

You had a clue in their response:

compensating controls

This phrase is used to justify leaving potentially breaking code in place. The compensation could be anything, but whatever it is limits the scope of the exploit in such a way that it isn't an SLA-threatening problem. Perhaps the exploitable leak causes a brief outage in a small percentage of overall users, which in the grand scheme of their SLA monitoring gets lost in the statistical noise.

It is also entirely possible that they've misunderstood the magnitude of what you've found. Unfortunately, only they can judge whether or not their compensating controls are sufficient.

As the reporter, your role here is to provide them with a clear assessment of the risks this presents. You will be more effective in changing their perception of the risks if you can provide a realistic scenario where this weakness can be leveraged to a wide-spread outage. Be specific, try not to invoke mega-scale botnets, and a reasonable assessment of how easy the problem would be to exploit once the weakness is known it exist. Use terminology from the Common Vulnerability Scoring System; or better yet, cite any CVE numbers for the problem.

|improve this answer
  • Without knowing what are the controls, I can’t say much. It seems nothing can prevent me from doing it on many several servers. They already know that it takes five minutes to crash a sever during several hours. I guess making it effective on a larger scale would bring to jail. – user2284570 Nov 25 '15 at 22:50
2

You can lead a horse to water, but you can't make it drink.

If you were hired for a security audit, you already did what you could do by reporting the problem. If you were not paid for this, you already did more than you were obliged to do, and all you can do is avoid using that service in the future.

If the management decides that the risk is not large enough to justify action, that's their decision.

|improve this answer
  • In fact I now understand the “controls” concern account creation since it can only be perform with push requests, and that I was wrong about the clone requests. In the meantime, I discovered an exploitable heap overflow on GitHub. So I won’t investigate after that question since my bank accounts would fall into permanent taxation If I win more money this year. – user2284570 Dec 6 '15 at 22:58

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.