I would put this on the same level as installing mysqladmin or phpmyadmin on a public web server. Ideally you would not put applications that can do administrative actions on a publicly accessible location (even when protected with passwords, you are vulnerable to a brute force attack).
Furthermore, Tomcat's manager has been known to have had security flaws, so I would avoid using it from anywhere but localhost.
If I understand you correctly, you are trying to access Tomcat's manager from your office. In that case my recommendation would be set it up behind SSL, and to require an approved client SSL certificate for access - this should make it secure enough, provided that you trust your office workstation.