I'd put it a slightly different way to the other answers: proprietary vs. open doesn't make a difference to the actual security of the system, and framing the question in that way detracts from any argument you make.

SSLv3 is an open standard, but it's broken and vulnerable. If I go build a protocol that uses pinned mutual ECDSA authenticity, ECDHE key exchange, and ChaCha20-Poly1305 as a bulk AED, but didn't document it anywhere, it'd be proprietary but more secure than SSLv3 by a long shot.

TLS has long been considered a secure protocol that offers a high level of security assurance, due to the large number of qualified experts who have put work into analysing it. Nobody ever batted an eyelid when using OpenSSL, because TLS is well understood and secure. In 2012, the Heartbleed vulnerability came as a wake-up call to this line of thinking. As it turned out, nobody had been looking. OpenSSL was being maintained by a small, under-funded group, and because everyone assumed that it was secure and that qualified people were auditing the code, few security people looked for bugs. The de-facto library for the world's most-used secure protocol was full of vulnerabilities.

An application being open-source doesn't imply that it is secure. For many years the "many eyes" principle was touted as being a greater guarantee of security, but in practice it has shown not to be true - open source software produces vulnerabilities at the same rate as any other software, and the time between introduction and discovery of those vulnerabilities is not measurably shorter for open source software.

Security professionals are more than capable of analysing closed-source proprietary software packages and their network communications. Many of us built a career on it. Having access to the source is helpful, but it is not mandatory. In some cases (e.g. .NET applications) assessing the binary actually ends up being more convenient, because the decompiler software offers better code-browsing features than the source files do alone.

What this really comes down to is how much security analysis has been performed on the system in question, and whether that level of analysis is commensurate to the system's size and complexity. This goes for both the protocol and any implementations of it.

This is why the way you frame the question is critical when it comes to developing a threat model and arguments for or against a particular option. If you frame the argument as "open" vs. "proprietary", you'll end up in a philosophical debate that doesn't serve to answer the real questions.

A better line of questioning is:

1. Does the solution being evaluated come with an established level of security assurance?
2. If the level of security assurance is unknown, what are the costs associated with ascertaining the security posture of the solution?
3. If there are costs involved with evaluating the security of the solution and attaining sufficient assurance, how do they change the cost/benefit analysis of the solutions being considered for this business requirement?
4. Does the level of security assurance you have (or can attain) provide sufficient risk reduction in the context of the system? (i.e. is the risk acceptable?)

Established standards with well-understood levels of security assurance are useful because they often allow you to mostly skip questions 2 and 3. However, that doesn't mean that they're the best option in every case - protocol security is only part of the business decision when it comes to choosing a solution to meet a business demand. That's why question 4 is important: you're not trying to make the system as secure as possible, you're trying to reduce risk to an acceptable level. Framing the question around assurance, risk, and cost/benefit allows you to make informed security decisions that serve the business' goals without getting bogged down in security absolutism.