Should I really treat memory leaks like classical bandwith dos?

Question

I recently found that a big company use an outdated product in their file-storage servers (I got the hidden name and version). The bug is corrected in recent version, but they weren’t any cve or exploit until I create them. (their is still no cve and many distributions still sheep affected git packages in current branches)

They do set limits on ram usage on their http backends but not on their virtual storage servers (a single process can stall the server on swaps)
They do per account load balancing which means each storage server contains the entire data of http accounts (they have hundreds thousands accounts). I was unable to find how many they have, but considering their site is a top web site (in terms of worldwide traffic) I just guess this should be a lot.

I tried the issue and finally crashed one of their storage server during several hours with a single crafted request (the http server was sending 500 pages to the users trying to access the accounts stored on it)
I apologised and mailed them about the behaviour. I raised the concern of a larger scale attack that would shut down all file servers leaded by a team of 5~10 peoples without requiring botnets.

Hello again,

In these cases we feel that our compensating controls can make this significantly harder to exploit. You are probably correct that all of these things are possible and ideally would be fixed, but we consider this to be very low risk. I'm very impressed with the investigations you have done !

They really seems to treat this like classical attacks from botnets affecting bandwidth.

I learned to always update software that suffer from exploitable memory leaks. However I’m only a student, I have no experience at managing such a large website (~47 valid requests per minutes). But here are some points :

• According to their status page, none of the classical dos attacks they faced impacted users (which isn’t the case here)
• While their “controls” might prevent the whole site to be put offline, it definitely won’t prevent a large downtime of several of their storage servers (affecting thousands users).
• The purpose of the site is to host software, it’s perfectly possible to target the most downloaded ones.
• Since storage servers have 50Gb of ram I don’t think they have so much of them (I don’t know if large needs like this can require a >20Tb of ram)
• It can only be used for dos. There is no longer possibility for things like remote code execution.
• Once a storage server recovered, it’s possible to put it down again immediately since nothing prevent a crafted request to be made from a different machine with a different ip and it don’t require to log‑in (then the repository is unavailable most of the time for several days)
• In the meantime, they do many things to prevent a push request from filling ram, like limit the maximum object size, or disallow deep nested tree objects. They also took actions on a prior problem allowing ram exhaustion from one of my report.

Something I can’t understand is they worry about non‑root remote code execution on a single of those server whereas the software that power them is public (they don’t worry about the fact a large part of their configuration is accessible from outside).

So are there situations where keeping exploitable memory leaks is Ok ? (only 1.2Gb of network data is required for stalling a 50Gb server)
Or more generally, should ram be limited on ɢɪᴛ servers ?

Answer 1

The answer to this comes down to a risk-management decision that the organization makes. There are best practices, but even the bestest of practices can be compromised by risk-management decisions. There are no absolutes, even patch broken stuff is not an absolute.

You had a clue in their response:

compensating controls

This phrase is used to justify leaving potentially breaking code in place. The compensation could be anything, but whatever it is limits the scope of the exploit in such a way that it isn't an SLA-threatening problem. Perhaps the exploitable leak causes a brief outage in a small percentage of overall users, which in the grand scheme of their SLA monitoring gets lost in the statistical noise.

It is also entirely possible that they've misunderstood the magnitude of what you've found. Unfortunately, only they can judge whether or not their compensating controls are sufficient.

---

As the reporter, your role here is to provide them with a clear assessment of the risks this presents. You will be more effective in changing their perception of the risks if you can provide a realistic scenario where this weakness can be leveraged to a wide-spread outage. Be specific, try not to invoke mega-scale botnets, and a reasonable assessment of how easy the problem would be to exploit once the weakness is known it exist. Use terminology from the Common Vulnerability Scoring System; or better yet, cite any CVE numbers for the problem.

Answer 2

You can lead a horse to water, but you can't make it drink.

If you were hired for a security audit, you already did what you could do by reporting the problem. If you were not paid for this, you already did more than you were obliged to do, and all you can do is avoid using that service in the future.

If the management decides that the risk is not large enough to justify action, that's their decision.