So, first things first: the term "end to end encryption" refers to something that does not generally apply to TLS. End-to-end encryption would be if, for example, I encrypted this post such that only you could read it, and StackExchange could not. After all, the primary communication here isn't between me and the StackExchange server, it's between me and you (for arbitrary values of "you"); the server is merely relaying the communication, rather than acting as an endpoint. As such, even though StackExchange uses TLS for all its external traffic, no portion of it is end-to-end encrypted.
TLS encrypts (and provides other protections such as integrity - no bit flips or data reordering - and replay protection) traffic between two sockets, generally owned by processes running on different machines (e.g. your web browser and StackExchange's web server). It also allows the client (the party initiating the communication) to verify the identity ("authenticate") of the server... and optionally, for the server to authenticate the client as well (this is almost never used on the public web, though it can be; your browser supports it). All communication between those two hosts is opaque; all that a middleman can see is the amount of traffic flowing each way, where it's coming from and going to, and the timing of the transmissions.
And there are middlemen, even if there's no actual man-in-the-middle attacker. Your TLS traffic travels through a bunch of devices including your home router, your modem, your ISP's modem and gateway, the Internet backbone routers, the server's ISP gateway, the datacenter router, and then finally probably TLS gets terminated at a load balancer before your traffic is forwarded - possibly inside a new TLS session - to the actual web server host. Every single one of those modems, gateways, and routers could in theory read and modify your traffic... but TLS prevents them from doing so successfully. The only party who can do that is the one that the client considers to be the server, identified using a certificate and proven using a corresponding private key.
This is potentially true even if you use a proxy! A proxy might be configured to terminate TLS - in which case it needs to be able to convince your browser (or any other client) that it's actually the destination server - or it might just relay the TLS-encrypted traffic without any ability to peek at it. With that said, TLS-terminating proxies are used for lots of stuff; your employer might have one that monitors Internet use from within the company network, and servers might have one (sometimes called a "reverse proxy") to provide DDoS protection or other services (this is how Cloudflare works).
Hopefully this clears up your confusion. It's not about whether there is a middleman - there almost always will be, unless you're communicating over local machine loopback and not using the network - it's about whether the client (e.g. browser) considers any of those middlemen to be the destination (server). The connection between the client and what it considers to be the server (which might actually just be a proxy or load balancer or similar, if those have the server's certificate and private key) is opaque to everybody else. However, it's not generally considered end-to-end encryption, because the server usually isn't the final destination of that traffic.
To answer your last question about whether you should use application-layer encryption (which - technically speaking - TLS is, but I know what you mean), the answer is almost certainly "no", but it depends on your use case. There's no reason to worry about TLS itself being breached, respectfully, if you had to ask this question, there's no chance you will implement a protocol that can withstand anybody who figures out how to breach TLS. However, if you want end-to-end encryption where even the server can't read the traffic, that's where adding another encryption layer comes in. Consider apps like Signal or WhatsApp; they use TLS between the client and server, but the actual messages are end-to-end encrypted such that the server can't read them, only the final recipient(s). If you're making something like that, then maybe you want to also use something more than TLS. For all other purposes, don't bother; it won't add anything except wasted effort and headaches.
