Here is a setup:
A Linux/Windows VM at home (will call it "hidden VM" from now on) which is placed into an isolated environment (it's located in its own isolated virtual network, and the only way out from there goes through another VM where a Zeroshell software router is installed; one of the Zeroshell VMadapters is bridged into a real physical network (a home lan, which is then connected to the internet through a physical router device, which also performs NAT transformations "home lan -> Internet"), other "looks into" mentioned isolated virtual network; Zeroshell box also performs NAT transformations "isolated virtual network -> home lan"; moreover, Zeroshell box drops all connections except for those directed to the ip address and port of the VPN server)
VPN server is located on a rented VPS (Ubuntu 14.04 is installed there), OpenVPN runs there on udp port 1194. VPN server does its own NAT transformations "private internal network -> Internet", and it uses ip address different from the one that it listens for client's connections on as an external ip address for NAT (so there is no way target server could port scan that ip and discover there is open udp port 1194 where VPN server listens)
On hidden VM DNS server setting manually set to 8.8.8.8 (to remove any possibilities of leaking the real external ip address through DNS requests to local DNS servers).
On the hidden VM wireshark is running, which is configured to show only those packets that are not coming in/out from/to port 1194 of the VPN server, to make sure that all communications are indeed going on through the VPN tunnel.
Now here is the issue:
There is one site that makes heavy use of Adobe Flash. Even the login form is implemented with it. This site works fine in the hidden VM when the VPN tunnel is down. It also works fine when you are accessing it from the VPS box where the OpenVPN server is installed (like, if you install a DE there and connect to it via VNC; I tested it to ensure that VPN server's address doesn't end up in some kind of black list after VPN tunnel presence was detected), but it stops working in hidden VM when you create a tunnel to the VPN server from it, to hide your real external IP address behind the VPN server's IP address.
The weird thing is how this "stop working" appears to happen. The UI looks the same as before. You can input your credentials and click the "log in" button, the login form then slightly changes its look - and nothing happens. If you provide an incorrect password, it notifies you about that. However the odd behavior doesn't end there. If you connect to it with when your tunnel is up by using a freshly obtained new IP address on your VPN server, it MAY allow you to create a new account, and MAY allow you to use the site a little, but eventually when you try to access the site again one day it will exhibit the same aforementioned behavior again, and it will be broken from that moment on anytime you try to access it through a tunnel (that's why I had idea VPN server's ip address may become blacklisted at some point). Kill the tunnel - and all is fine right away, set the tunnel - and it's broken.
How can they do that taking into account the precautions I have taken? And do they do something at all (i.e., may be it's just a bug in their codes?) As far as I know Flash/Action Script and Javascript (seems nothing else aside from those is running on the page) are pretty high level technologies, so there is no way they can find out my real external IP address from within the hidden VM (as they would be unable to override the default route set by the tunneling client, neither they would be able to directly inspect the OS running on a hidden VM, to find some clues about it), thus finding out that it differs from the one they see me connecting from. I believe there is no way it can start some java applets without my notice, even if it's signed it MUST present some kind of notification window, so there shouldn't be a way for page's content to escape the jail.
Wireshark doesn't show anything except for ARP traffic and some strange rare retransmitted packets. I even tried to set my clock so it corresponded to the timezone of the region where my VPN server is located, still no luck.
Does this ring a bell? Is this a bug in their code, and if so, then what can it possibly be? Is there any way they can detect I'm using a VPN tunnel?
Edit: I've attached a network topology diagram to help understand my setup a bit.
Edit2: Just a quick notice (as there is too much text so the point may be hard to spot): the main goal of this research is to understand whether the cause of this issue is the fact the target site is able to detect that I'm using a VPN solution, or not, and if it is, then find a fix for the solution so it couldn't detect it anymore. So I'm trying to come up with ideas on how it can possibly detect it, despite all my precautions.
Edit3: I've got a word that they indeed are using some VPN-busting solution, though no idea why it works like that, instead of simply telling you that you are denied access because you are using it.
