In the United States, any system that handles a certain volume of customer payment method details (credit/debit card numbers, bank account numbers, etc) must be PCI compliant to operate legally. If the system you describe has your credit/banking details and allows you to view them through that web interface, you may have valid cause to report them for non-compliance.
Alternately, if the website stores certain details or a combination of details regarding your personal identity, they may be required to follow other regulations pertaining to the storage of Personally Identifiable Information (PII). In some cases, even just your first and last name together can be considered PII. (Example: The name "John Smith" stored separate from any other personal details is not PII, but "Joachim Schlichenmeier"* would be.)
I'm not personally aware of the procedures for reporting such violations to any entity that is capable of acting upon them, so for that I would suggest you consult with an attorney. Of course, you'll have a much better case if you at some point are subject to identity theft and the root cause can be traced to your utility company's bad practices.
Beyond that though, I suggest you follow the recommendations others have posted here. Use long and complex passwords for all websites & applications, and do not re-use passwords across any websites & applications. Additionally, limit the information you give these websites & applications to only that which is absolutely necessary for them to serve their purpose.
If there really is no need for the site to permanently store your credit card details, leave that little "remember this" box un-checked. Especially follow this for places you know are un-trustworthy, like your utility company. If you can make payments over the phone or via postal mail, I would suggest pulling all of your payment info off of your profile on the website (if it's there at all) and switching to one of the old-school methods.
*Note: The name "Joachim Schlichenmeier" is intended to be fictional. Any relationship to an actual person, living or dead, is purely coincidental.