The security consequences of this server to you are likely negligible. Sending spoofed email is trivial no matter what you do. Even if you lock down or turn off this server, it will still be simple for anyone to send a spoofed email with a spoofed From: address. You can't stop it by tightening up this server, since attackers are free to send their email through any SMTP server you want, and there are too many other open relays in the world to stop this.
The consequences to the rest of the world are that you are arguably contributing, by a tiny amount, to the ease of spam. You are running an open relay. Open relays let any spammer send spam with a spoofed From: address. For these reasons, open relays are considered a bit anti-social and generally frowned upon.
It's a bit like polluting: if you pollute just a little bit, the risk to your own health may be minor, but if everyone did that, our skies would be a horrid haze of pollutants, and everyone would suffer. Therefore, for everyone's sake, the right thing to do is to avoid running an open relay.
The biggest risk to you is that spam-fighters may detect that your SMTP server is acting as an open relay and blacklist your server as a result. This could cause email you send to others to be marked as spam by the recipient email system. For these reasons, you probably don't want to keep your SMTP server available as an open relay.
You can test whether your mail server is configured as an open relay using any number of online testers.
I'm not sure what you mean when you talk about sending an email through this server without any kind of actual spoofing. I'm not sure what distinction you are drawing. Connecting to a mail server, pretending to be Alice, and asking it to send email from Alice is exactly spoofing. In other words, what you're talking about is exactly how email spoofing is done.
