I recently found that a big company use an outdated product in their file-storage servers (I got the hidden name and version). The bug is corrected in recent version, but they weren’t any cve or exploit until I create them. (their is still no cve and many distributions still sheep affected git packages in current branches)
They do set limits on ram usage on their http backends but not on their virtual storage servers (a single process can stall the server on swaps)
They do per account load balancing which means each storage server contains the entire data of http accounts (they have hundreds thousands accounts). I was unable to find how many they have, but considering their site is a top web site (in terms of worldwide traffic) I just guess this should be a lot.
I tried the issue and finally crashed one of their storage server during several hours with a single crafted request (the http server was sending 500 pages to the users trying to access the accounts stored on it)
I apologised and mailed them about the behaviour. I raised the concern of a larger scale attack that would shut down all file servers leaded by a team of 5~10 peoples without requiring botnets.
Hello again,
In these cases we feel that our compensating controls can make this significantly harder to exploit. You are probably correct that all of these things are possible and ideally would be fixed, but we consider this to be very low risk. I'm very impressed with the investigations you have done !
They really seems to treat this like classical attacks from botnets affecting bandwidth.
I learned to always update software that suffer from exploitable memory leaks. However I’m only a student, I have no experience at managing such a large website (~47 valid requests per minutes). But here are some points :
- According to their status page, none of the classical dos attacks they faced impacted users (which isn’t the case here)
- While their “controls” might prevent the whole site to be put offline, it definitely won’t prevent a large downtime of several of their storage servers (affecting thousands users).
- The purpose of the site is to host software, it’s perfectly possible to target the most downloaded ones.
- Since storage servers have 50Gb of ram I don’t think they have so much of them (I don’t know if large needs like this can require a >20Tb of ram)
- It can only be used for dos. There is no longer possibility for things like remote code execution.
- Once a storage server recovered, it’s possible to put it down again immediately since nothing prevent a crafted request to be made from a different machine with a different ip and it don’t require to log‑in (then the repository is unavailable most of the time for several days)
- In the meantime, they do many things to prevent a push request from filling ram, like limit the maximum object size, or disallow deep nested tree objects. They also took actions on a prior problem allowing ram exhaustion from one of my report.
Something I can’t understand is they worry about non‑root remote code execution on a single of those server whereas the software that power them is public (they don’t worry about the fact a large part of their configuration is accessible from outside).
So are there situations where keeping exploitable memory leaks is Ok ? (only 1.2Gb of network data is required for stalling a 50Gb server)
Or more generally, should ram be limited on ɢɪᴛ servers ?