If you cannot (at least easily) switch providers (I assume that you are locked in to a service which cannot be moved) then you have to have a few things covered. For you, your business and your customers (if they make use of the service).
To start with, I am not a lawyer so you will be better off discussing this with a lawyer.
First read your service contract: is there anything about "reasonable efforts" the provider is supposed to make, from a security standpoint? If so, there are industrial best practices about how an application should be secured. You could point out to your provider that he is possibly in a breach of agreement.
Now - he may not care, knowing that you are locked in with him and resolving the issue in court is too long/complicated/whatever to you.
In that case you need to do some due diligence. The idea is that, in case of bad things happening (the service is breached, your and your customers' data is in the wild) you can prove that you have done all reasonable steps to alert about possible issues. Keep a copy of the exchanges. Then warn your admin that in case there is an issue, he will be responsible for all costs.
What you can do and the true value of your actions will considerably vary with the exact circumstances of your case. Again - consult a lawyer.
I would also strongly recommend planning for a provider switch. I am very much aware that this is not an easy option but it usually boils down to resources (time, money).
Finally, do not make any attempts to hack his site for demonstration purposes. This is not only possibly illegal (and subject to appreciation of a jury) but you may also harm his service with all the consequences (interruption for other customers for instance).
