Is this TCP socket encryption and password storage implementations secure enough?

Question

I'd like to know if this communication technique is secure enough against attacks, both on the connection itself and the password-storing procedures.

• Both the client and the server have access to 2 pre-shared static keys:

  Enc Key A (32-byte string)

  Enc Key B (32-byte string)

• The server has access to a static key:

  DB Enc Key

Then I proceed as follows:

1. Client connects to a server (regular TCP connection)

2. Server generates a crypto-safe ECDH key pair (public and private keys), using SynCrypto Library (https://github.com/synopse/mORMot/blob/master/SynCrypto.pas)

3. Server encrypts (AES-256 bit with Internal Salt) the public key using "Enc Key A" as the secret and sends it to the client

4. Client generates its own key-pair, then also encrypts its public key using the "Enc Key A" as the secret and sends it to the server

5. Both client and server generate a shared secret (SynCrypto) and hash the [Enc Key B+shared secret] 1000 times (HMAC SHA-256) using "Enc Key A" as the secret

   r = Shared_Secret; repeat(1000) r = hmac_sha256(Enc_Key_A,Enc_Key_B+r);

6. All further communication between server and client will be encrypted (AES-256 bit with Internal Salt) using the hash as the secret

As for the password storage, for each user (once for each user, not once for every connection) I generate a unique key, which is then used to hash the password 1000 times:

r = Password;
repeat(1000) r = hmac_sha256(DB_Enc_Key,UniqueKey+r);

Then both the UniqueKey and the "r" (hashed password) are stored in a plain text file

I'm using GML (GameMaker Language) for both the server and client and using a Pascal .dll for the cryptographic functions.

• Is that secure enough?

• Is it vulnerable to attacks?

• How can I improve its security?

Answer 1

Is it vulnerable to attacks?

1. Your scheme is vulnerable to replay attacks.
2. If you use the same key for all clients, leakage of the key at one of clients will allow man-in-the-middle attacks, it will allow to decrypt and to modify the traffic of all clients.
3. You don't describe how are you going to organize sharing of the keys. Depending how you do that it can be a further security problem.
4. You don't describe how are you going to prohibit some key in case one of clients believes the key was stolen and wants to use a new key. Depending how you do that it can be a further security problem.

How can I improve its security?

Use TLS, as @SteffenUlrich suggested.

Alternatives

Since you say you use some DLL on the client side, I suppose your clients run on Windows. If GML does not provide TLS support, I'd suggest you implement communication layer based on TLS with some other tools, e.g. with C++. Then there will be no need to implement TLS, you will just use it. Implement a high level communication layer over TLS, put this code into DLL and use it from GML. Thus you will get communication over TLS.