What is the mechanism used by websites like HBO GO to detect HDCP bypassing/violation? How foolproof is this mechanism, as the logic lies on the client side?
It really depends on the device where the app is running (smartphone/tablet, set-top-box, smart TV...) but typically the detection happens in code running in a CPU mode (and using reserved memory) that cannot be accessed by regular apps. The most typical implementation is by means of a Trusted Execution Environment (TEE).
Before transmitting High Definition content, the transmitter initiates authentication process with receiver and exchanges the KSV. Basically it is having three level of authentication. 1. Authentication between transmitter & receiver. 2, if receiver is repeater, additional authentication, 3 during data transfer every blanking interval.