I think there are two fundamental misunderstandings here (or perhaps two statements of the same misunderstanding):
From this we can derive that web browsers allow remote code access to many if not all functions of the device.
Remote code execution doesn't imply that the application has allowed access, only that it has failed to prevent it in some way. Consider a burglar breaking into your house: you didn't allow them access, but they found a way to gain that access, against your wishes; maybe they spotted a window that was slightly loose in its frame, or a catch that could be reached from the letterbox.
it seems to me it would be easier to build software that gives remote access to nothing outside of the page that is being displayed, than to give remote code access to everything, and then try and lock it down
At the most fundamental level, all code running on a piece of hardware has access to that hardware; the only reason we can talk about "granting access" at all is that there are many layers of abstraction "locking down" that access. Taking the burglar example again, if your house is just a line on the ground, anybody can walk in; it is only by building walls that you can even hope to keep them out.
There is a grain of truth in what you say, though: if you provide a very small set of possible operations, the scope for evading the intended security is much smaller. If you have a house with only one door and no windows, you only need to defend that door against burglars - it might not be a very pleasant house to live in though. Similarly, you could build something resembling a web browser which was the only thing running on the computer (no OS, no other applications), and displayed only plain text (no styling, no interactivity) - good luck marketing that one.
The reality is that modern browsers are extremely complex pieces of software, running on top of and alongside other complex pieces of software (operating system components, applications for handling downloaded files, etc). Even though every piece is intended to restrict access in many ways, there are lots of places that attackers can probe. They are less like windowless bunkers with only one entrance, and more like grand palaces with hundreds of doors, windows, roofs, and chimneys, and the occasional loose brick that you can pry out if you spot the cracked cement.