My guess is that it has a serious vulnerability that could be exploited, but it all depends on the implementation. Saying it uses RSA tells very little about the effective security of the system. RSA will not be the weakest link.
- Does it prevent replay attacks?
- Does it provide Authenticated Encryption by using MACs?
- Does the client verify the identity of the server at the other end before responding to challenges?
- After your login could an attacker alter your remote commands to do malicious things?
You could try browsing the source code (ServerPAMAuth seems a good place to start) and try finding flaws, but honestly I wouldn't bother.
What I would do, is set up a reverse proxy that uses SSL. It just happens I wrote up a detailed guide for how to do that in nginx for a webcam the other day -- just replace webcam with RStudio server and you'll be fine, and note you can run the proxy and Rstudio on the same server (so replace references to
127.0.0.1 for best results). Then even if the authentication protocol is vulnerable, you'll still be protected (assuming you actually use a CA signed certificate, or get your browser/other application to trust your self-signed certificate).
EDIT: Actually, according to their instructions you'll need to add a proxy_redirect line for:
proxy_redirect http://localhost:8787/ $scheme://$host:$server_port/;
This says if you have a full URL in an HTTP header field (e.g., in a 302/304 HTTP redirect where the Location field is a URL telling where the resource moved) it will change
$scheme://$host:$server_port/path/to/something as necessary, where
$scheme will be