I’m planning on setting up a network with a custom-built router with pfsense. If I assume the cable modem connected to the router is compromised can a properly configured router ensure security or do I need to turn my custom router into a custom cable modem? I’m only concerned about my data sent and received through the modem being read unencrypted or the modem having access to my LANs on my router.
2 Answers
A compromised modem would only be able to compromise the router if it can exploit some vulnerability in the services that are exposed to it (which should be minimal, because there's no reason to expose the administrative interface on the WAN side of the router). Alternatively, if the router did anything insecure like installing updates of unencrypted HTTP, then those could be intercepted and modified (using something like evilgrade.
However, it will be able to intercept and modify any traffic that flows though it. If all that traffic goes down a properly configured and encrypted VPN tunnel then there would be fairly limited scope for attacks (although it could still gather metadata, or cause a denial of service).
It's basically the same position as having an untrusted ISP or being connected to a dodgy wireless network - as long all your traffic is well encrypted and you have mutual authentication, it's hard for them to directly compromise you.
If the modem is a separate device being connected via Ethernet or some serial link, then Gh0stFish's answer holds (all the modem could do is tamper with unencrypted traffic or exploit a vulnerability in some protocol parser), but if the modem is in the form of a PCI or PCIe card in the router (which is very common with modem routers), then all bets are off. Most routers do not have the hardware necessary to protect against DMA attacks by devices on the PCI bus, and a PCI modem may be compromised through a malicious firmware update using TR-069.
If the modem is connected using Ethernet, PPP, or something similar and the traffic is encrypted and authenticated (i.e. not vulnerable to MITM attacks) and the router's network drivers are not seriously outdated and vulnerable, you should be fine.