Can proprietary protocols be considered as secured?

Question

I am new to the info-sec industry and was recently tasked with evaluating the communication protocols used by my various subsystems.

So it was stated in the requirements that the systems have to implement secure communication protocols and secure file transfer mechanisms to safeguard data confidentiality in transit over the network.

The issue is some of the systems have been telling me that they can fulfil this requirement by using proprietary protocols. I personally do not agree with this statement as it is very debatable.

Any suggestions on how I can/should voice out my concerns in a way to bring my point across?

Also, since it was mentioned in the requirement to safeguard data confidentiality, can it be assumed that all other matters like integrity and availability can be taken out of consideration when proposing solutions?

Answer 1

Secure communication protocols

The one question that gets overlooked too often in the infosec industry is "secure against what?" The phrase "I want to be secure!" is like saying "I want to be happy!" It could mean anything in any context, and you won't get secure or happy unless you define it.

So, the requirement to "implement secure communication protocols" needs definition. What makes a protocol secure? How does one know that one has implemented a secure protocol? What does the protocol need to be secure against? Once that is defined, then an analysis against requirements can be carried out. Else you end up in a battle of personal opinions.

You mention "safeguard data confidentiality" but seem to be confused about other requirements. Other requirements need to be defined.

So, before going back to the system owners, make sure you have a defined requirement.

Proprietary protocols

The response from system owners has a name; "security by obscurity". Since there is no guide or manual to the protocol that someone outside of the system could learn from, then someone would need to learn how the protocol works before exploiting any weakness.

"security by obscurity" has its uses, but never as a security control. It's like hiding your front door key under the mat. It's perfectly hidden ... until someone finds it and then all security fails.

So, the system owners need to define how their protocol secures against a breach of data confidentiality (the one requirement you mention, there might be more). "We use our own protocol" isn't an answer to that question.

Answer 2

"The issue is some of the systems have been telling me that they can fulfil this requirement by using proprietary protocols. I personally do not agree with this statement as it is very debatable.

Any suggestions on how I can/should voice out my concerns in a way to bring my point across?"

Ask them to provide a formal proof of it. "I can't think of a way to break this, so no nobody else will either" obviously isn't good enough, but if they're really expert enough to implement a secure protocol, then they ought to be able to justify their confidence. How did they test it?

You could also possibly find some examples of protocols used elsewhere that have been broken, present them, and ask them to point out which have flaws and what they are. If they can't do so, then that ought to convince them. If they can, then maybe that would reassure you. Something like this is probably too simple, but demonstrates the idea. (If you ask, maybe somebody has a better list.)

Perhaps you could look up some examples of recently discovered exploits and ask them about them. If they're security professionals, they ought to be keeping up with developments in the field. Feel out their knowledge. Again, if they are obviously well informed about them, you can relax. If they don't know what you're talking about, that might clue them in to being out of their depth.

And from a talking-to-management point of view, ask about the required qualifications and experience they demand for people doing security design. There are a whole bunch of formal qualifications and exams for this sort of thing. I'd be suspicious about whether they've got any, since I think most courses would emphasise the dictum "Never roll your own crypto", but you never know. And ask whether their protocols have been through any formal accreditation process like Common Criteria (ISO/IEC 15408). If they don't know what you're talking about, that's another red flag that they don't know what they're doing.

There are people working for private companies who do know what they're doing. Plenty of top cryptographers work in industry. And if they have a big enough group of big enough names working there, then they may well be able to give it a very thorough review internally. Don't rule it out as impossible. But it's unusual, and usually you would know about it if that was the case. When I did crypto, they told me that you should first have established a track record of having broken lots of other people's cryptosystems before you even think of designing your own. The same goes for protocols, I expect. So have they?

And I'd collate a long list of press cuttings about companies that implemented their own protocols, got hacked because of undetected flaws in them, and lost millions or generated horrible publicity from having their customers' personal details sold on the black market. They all thought they were safe too. What makes them so sure they're different?

Answer 3

This is close to an opinion debate. What matters here is trust. A public protocol that has been scrutinized by experts in a specific implementation is generally seen as secure. And most security experts know that peer reviews are the possible best to identify logical or implementation flaws.

But some organizations say that they have enough experts to be able to process those peer reviews. If you already trust them (whatever the reason) for mission critical things you can trust them on that point too. Or if you are working for a governmental organization, and the IT security service of that government advises you with a closed protocol, you can use it. Past that point, reputation can matter.

Of course I would not trust a small or recent company to have invented the best protocol and do not want to share it. Because it often means that they are not expert enough. But I would not trust more a company that would use a public protocol with a non standard implementation even if they provide the source, because I know that only true experts can scrutinize it.

TL/DR: the key is not public/private protocol. It is more what reputable sources say that the protocol and the concrete implementation has been reviewed.

---

I have insisted on the implementation point, because flaws have been found in implementations of otherwise correct protocols...

Answer 4

I'd put it a slightly different way to the other answers: proprietary vs. open doesn't make a difference to the actual security of the system, and framing the question in that way detracts from any argument you make.

SSLv3 is an open standard, but it's broken and vulnerable. If I go build a protocol that uses pinned mutual ECDSA authenticity, ECDHE key exchange, and ChaCha20-Poly1305 as a bulk AED, but didn't document it anywhere, it'd be proprietary but more secure than SSLv3 by a long shot.

TLS has long been considered a secure protocol that offers a high level of security assurance, due to the large number of qualified experts who have put work into analysing it. Nobody ever batted an eyelid when using OpenSSL, because TLS is well understood and secure. In 2012, the Heartbleed vulnerability came as a wake-up call to this line of thinking. As it turned out, nobody had been looking. OpenSSL was being maintained by a small, under-funded group, and because everyone assumed that it was secure and that qualified people were auditing the code, few security people looked for bugs. The de-facto library for the world's most-used secure protocol was full of vulnerabilities.

An application being open-source doesn't imply that it is secure. For many years the "many eyes" principle was touted as being a greater guarantee of security, but in practice it has shown not to be true - open source software produces vulnerabilities at the same rate as any other software, and the time between introduction and discovery of those vulnerabilities is not measurably shorter for open source software.

Security professionals are more than capable of analysing closed-source proprietary software packages and their network communications. Many of us built a career on it. Having access to the source is helpful, but it is not mandatory. In some cases (e.g. .NET applications) assessing the binary actually ends up being more convenient, because the decompiler software offers better code-browsing features than the source files do alone.

What this really comes down to is how much security analysis has been performed on the system in question, and whether that level of analysis is commensurate to the system's size and complexity. This goes for both the protocol and any implementations of it.

This is why the way you frame the question is critical when it comes to developing a threat model and arguments for or against a particular option. If you frame the argument as "open" vs. "proprietary", you'll end up in a philosophical debate that doesn't serve to answer the real questions.

A better line of questioning is:

1. Does the solution being evaluated come with an established level of security assurance?
2. If the level of security assurance is unknown, what are the costs associated with ascertaining the security posture of the solution?
3. If there are costs involved with evaluating the security of the solution and attaining sufficient assurance, how do they change the cost/benefit analysis of the solutions being considered for this business requirement?
4. Does the level of security assurance you have (or can attain) provide sufficient risk reduction in the context of the system? (i.e. is the risk acceptable?)

These questions are useful because they come with answers you can either quantify or use to aid you in your decisions. They get down to the core issues of cost and risk. The process of getting answers to the questions also helps you build up a better understanding of whether or not the solution is a good fit for your organisation's needs from a security standpoint. You may even find that the vendor is willing to pay for a pentest on their own product, from a mutually agreed vendor, in order to make a sale.

Established standards with well-understood levels of security assurance are useful because they often allow you to mostly skip questions 2 and 3. However, that doesn't mean that they're the best option in every case - protocol security is only part of the business decision when it comes to choosing a solution to meet a business demand. That's why question 4 is important: you're not trying to make the system as secure as possible, you're trying to find a way for the business to achieve its goals in a way that maintains acceptable risk. Framing the discussion around assurance, risk, and cost/benefit allows you to make informed security decisions that serve the business' goals without getting bogged down in philosophical arguments or security absolutism.

Answer 5

"Proprietary" can mean a lot of things.

In InfoSec, the important question is how well something has been examined. That is where most proprietary protocols fail. They aren't used widely, have not been examined by many independent experts, and as such while they are not necessarily less secure we know less about their security than with protocols that everyone and their dog has been hitting on for the past three decades.

In practical terms, for any proprietary protocol I would see the burden of proof with the proprietor. And I mean "proof". I don't mean anecdotal evidence ("ten people use it and nobody has been hacked to far!"), or argument from authority ("we are experts, don't you trust us?" - footnote: The answer to that is always no. Trust in InfoSec has a different meaning and is not based on human factors).

I know of a few proprietary protocols that do have formal proofs and have been subjected to intense scrutiny by independent experts. They are the exception. Most proprietary protocols are shitty.

You don't go into much details, so I need to speculate a bit. You speak about data in transit and the necessity to protect its confidentiality. We do have something that does that. It's called TLS. The entire Internet runs on it. Why in the world would anyone invent a proprietary protocol for the same purpose? That is the question you should ask - there's a fire and people come with some chemical substance. The right question is: "What's wrong with water?" - there are some cases where you don't want to use water for good reasons, but most of the time, water works just fine and its side effects and disadvantages are well understood.

Same here.

---

Also, since it was mentioned in the requirement to safeguard data confidentiality, can it be assumed that all other matters like integrity and availability can be taken out of consideration when proposing solutions?

That's a completely different question. Formally, yes. In practical terms, confidentiality and integrity are closely related and if you can kill two birds with one stone, then why not? (and again, if some proprietary protocol only guarantees confidentiality, then why not use TLS which offers both C and I ?)

Answer 6

There is no such thing as a "secure communication protocol" - there are - at best - protocols that haven't been proven insecure yet.

The advantage of standard protocols is that you can expect that a lot of experts have scrutinized them - so:

1. obvious flaws are less likely
2. possible weakness/attack vectors are largely documented
3. non obvious flaws are more likely to get publicized before they impact you (chance for remediation)
4. you can rely on a small army of independent evaluations (trust in consensus)

Proprietary protocols are unlikely to have been tested and vetted as well. You generally can't examine them yourself. The organization and experts who are promoting them have a conflict of interest between disclosing vulnerabilities and selling their stuff. And even if you trust them, oversight by is more likely (fewer eyeballs - no consensus).

Their only benefit is that there is generally less of an incentive for attackers to look for exploits.

So ultimately, it's a matter of (limited) trust. You might be able to get away with something like "Protocol X is proprietary to company Y, so we cannot properly evaluate it. Company Y has documented their development practices and own evaluation there."

(Caveat for stacks - e.g. if what you're concerned is transport security and company Y has a custom protocol, but relies on vanilla TLS for transport, then you can still make a judgement based on that).

(2nd caveat for obvious vulnerabilities. You might know the details of the protocol, but if someone is trying to sell you that their proprietary variation on 3DES is secure they're out!)

Answer 7

When auditing the security of a system, then you do of course need to audit all the protocols. "This is a black box, so it's probably secure" isn't satisfying in a proper audit.

However, while "The protocol is proprietary" is not a proof of security, it is not proof of insecurity either. Yes, the security community is very distrusting of anyone who tries to do security by obscurity, which is why we want all protocols to be open. But fact is, whether the protocol is open or proprietary is irrelevant for its security. There are secure proprietary protocols, just as there are insecure open protocols.

But "Proprietary" also does not necessarily mean "impossible to audit". There are proprietary protocols which are indeed well-documented and specified. So when you can get a hold of the specification, then you can audit the protocol based on that specification.

If the specification is not available either, then another way to assess its security is by relying on a 3rd party. When a competent and trusted 3rd party got access to the protocol specification and implementation and certified the security of a protocol, then that might be sufficient for your case.

Answer 8

The Superior Security of One or the Other is a Cultural Bias, Not a Fact

Many security experts will claim that either Open Source or Proprietary is superior, but there have been shockingly few reliable studies to back this claim. Most of the articles you find on this topic are opinion pieces that cite other opinion pieces (if they cite anything at all).

Those articles that do make a strong, apparently research based claim all seem to use flawed methodologies. When making arguments for Open Source or Proprietary, most articles will cite the best examples of one thing they can find and compare it to any sampling of the other that seems to prove their point. So if you want to "prove" open source is better, you can safely compare the Linux Kernel and mySQL to any random sampling of proprietary software, and if you want to prove proprietary is more secure you could compare something like iOS to any random sampling of open source. This sort of cherry picking is how you manipulate statistics to produce whatever outcome you desire, and the Open Source/Proprietary debate is ripe with it.

The only research I can find on the topic that is not using an obviously flawed methodology is a 2005 study Software Vulnerabilities: Open Source versus Proprietary Software Security. This paper studied a random distribution of hundreds of open and closed source projects and found that there was no noticeable difference between how often Open Source and Proprietary software actually gets breached. The one actual advantage you do see with open source, is that they typically release patches faster when a breach does happen... but that said, proprietary software is more likely to include functional auto updating features; so, while Proprietary often takes longer to release a patch, Open Source is more likely to not get patched after it is released. Frankly, I think this study needs to be repeated to account for cultural development changes in the past 15 years, but I suspect the results will remain the same. This is because most of a system's final security hardening comes from post-release community feedback which follows more or less the same patterns regardless of open/closed source.

How you should be approaching this problem

Since there is no statistically significant difference, you can not use Open Source versus Proprietary as a reliable measure of security. In fact, you can never prove that any given system is secure. What you can do though is gather specific evidence that a system may be insecure.

Step 1: Look up a products CVEs.

Many products have publicly reported security vulnerabilities which are in some cases yet to be fixed. Many products have open CVEs that demonstrate vulnerabilities that developers have put on the back burner because they are not considered critical: either because they grant so little, or because they rely on rare implementations to exploit. If you find open CVEs that would be a problem to how you plan to implement the product, then you can stop here and safely say you need to pick another product.

Step 2: Look at product's community size & patch log.

While this is less conclusive than CVEs it can give you a good idea about how to judge the product's maturity.

A product's community size tells you a lot about how many opportunities a product has had to be exploited in the past. If a product has millions of downloads, then you can assume that many of those users should statistically have the skill set or random luck to have discovered and reported most, of the major security issues by now. But a product with a community size of just a few thousand could easily have a lot of yet to be discovered exploits.

A patch log can quickly give you a feel for how capable the developers are at getting things right the first time. If the log includes a lot of security patches within the past few months, that could indicate that the developers are doing a poor job of addressing security issues themselves before releasing updates. I would not trust such a product with anything important. Another bad pattern is to see no patches at all released within the past several months. This typically indicates that the developers have abandoned the project and are not fixing issues even if they are being reported. The most reassuring pattern is usually seeing a lot of security patches in a product's more distant past, and mostly Quality of Life updates in the recent past, because that tells you that the product has actually been subjected to real world implementation testing, and been patched until the community could not find any more exploitable vulnerabilities.

Step 3: Vulnerability & Penetration Testing:

This last part requires a bit more of a specialized background and tool set to do right; so, I would not consider it necessary unless your organization has particularly high cyber security goals.

Vulnerability testing is the scanning of an application for cases of "not best practices" that may be exploitable. Often times if a vulnerability scan comes up with a lot of issues, there is a good chance that at least one of those is exploitable, but you don't really know until you test it.

Penetration testing is the actual testing of exploits against a system. Sometimes, a system will fail a vulnerability scan, but not actually be exploitable in its implementation. So, the real proof of if a system is actually exploitable comes down to this. If a product fails a single penetration test, then like an open CVE, you have a factual piece of evidence that the product is insecure.

Conclusion

There is nothing wrong with being skeptical of any piece of software, but making a judgement call based on Open Source / Proprietary in lue of more measurably significant factors could leave you with a less secure business solution.

Answer 9

An important thing to recognize about most practical crytographic algorithms is that each of them isn't really a single algorithm, but rather recipe for converting an arbitrary pattern of 56 or more bits into one of of 2⁵⁶ or more independent cryptographic algorithms. If one generates a pattern of bits using some trustworthy method, and uses that method to "construct" an algorithm, the algorithm will be secure as long as the bit pattern is secure. If an insider who might have had access to the bit pattern leaves the company, selecting a new bit pattern via some secure method will effectively change the system to using a new algorithm which the previous insider wouldn't be able to break.

A propriety protocol designed by someone who is trusted and skilled may be secure as long as that person remains trusted, but if for whatever reason the person becomes untrusted it may be necessary to redo from scratch a lot of the work that had been done to make the algorithm secure against the now-untrustworthy insider. Designing an algorithm which would be reasonably secure for most purposes (*) against those who have no insider information about its workings isn't terribly difficult. What's hard is designing an algorithm which would remain secure even if people who would need to have had knowledge of its inner workings become untrustworthy.

(*) Secure enough that the effort required to break it would be greater than the value an attacker could receive by doing so.

In some cases, a proprietary algorithm may offer some real security advantages over commonplace ones. For example, there are some situations where it may be necessary to expose all of the information necessary to decrypt some information, but still exercise some effective control over how such decryption is performed. Code obfuscation can never achieve "real" security, but may be good enough to satisfy the aforementioned requirement. The effectiveness of obfuscation when using common algorithms may be undermined if an attacker can recognize and extract the key. Use of a proprietary algorithm may make it harder for an attacker to identify what parts of the program are actually necessary to perform the decrpyion.