RFC 6455 says:
Servers that are not intended to process input from any web page but
only for certain sites SHOULD verify the |Origin| field is an origin
they expect. If the origin indicated is unacceptable to the server,
then it SHOULD respond to the WebSocket handshake with a reply
containing HTTP 403 Forbidden status code.
The |Origin| header field [RFC6454] is used to protect against
unauthorized cross-origin use of a WebSocket server by scripts using
the WebSocket API in a web browser. The server is informed of the
script origin generating the WebSocket connection request. If the
server does not wish to accept connections from this origin, it can
choose to reject the connection by sending an appropriate HTTP error
code. This header field is sent by browser clients; for non-browser
clients, this header field may be sent if it makes sense in the
context of those clients.
The simplest fix is to check the HTTP origin header on the server side against a whitelist. Your library/framework may already provide secure defaults (no CORS) or have a way to configure this without requiring you to reinvent the wheel.
Since you do not expect any CORS requests, then you should verify that the HTTP origin header matches the URI of your website. According to the RFC, websocket upgrade requests should include this origin header. I confirmed on my own webapp that the header is sent even for the same origin.