If your security model is based around the concept of a point-in-time external audit of your entire codebase, then you're doing security wrong.
...And you're probably using the audit wrong too. But we'll get to that.
Beyond question, all code needs to be audited for security. In many cases, this is actually a legal requirement: no code ships without an audit, period. The traditional wisdom suggests that such an audit be an event at some point in the lifecycle, but a more sensible way to do it is to audit code as you go. That is, all code gets a security audit before it can be checked in to your codebase.
The theory is simple; the repository is already audited, so we don't need to re-audit its components as a standard procedure. But when a new feature or patch or bugfix is proposed, the diff has to be signed off by the appropriate maintainer(s). You can get sign-off for whatever is important to you. For example, the Linux kernel has a pretty involved approval process which requires several endorsements along the way for quality, simplicity, consistency, performance, etc. Your requirements may vary, but a security audit should be part of that approval process.
In this case you're not auditing the product end-to-end, you're just auditing the diff. But thousands of tiny audits over the course of the product's development cycle will be far more in-depth and comprehensive than any one end-to-end audit could hope to be.
A full-product end-to-end audit is certainly helpful and shouldn't be avoided. This audit should focus on the product as a whole in a way that isn't as easy to do during the patch-level audits you've been doing. You want to look at the whole forest from time-to-time, not just the individual trees. The timing of these large-scale audits should probably correspond to major releases, major changes, or compliance certification audits.
But by keeping current on the patch-level auditing, you can ensure that the code is always maintained in a verifiable state, so you can continue to ship on a regular basis with confidence.
About commit-time approvals
If your company isn't doing this, then you're doing everything wrong. There are dozens, hundreds perhaps, of problems that are solved by requiring every code change to be approved by at least one other person, including (and especially) during initial development. You should always have at least two people who understand how every line of code works, and who agree that the code is correct.
This is at least as important as unit tests. If you're not doing this, then stop everything and re-visit you policies around quality and security.
Yes, this process does scale. As noted above, the largest software project in the world uses it, as do some of the world's most agile and successful software companies.