Is this a secure hash "Function" for integrity?

Question

I was wondering what were the posible security (not efficieny) issues with this hash "function":

Although not an actual function, this method turns a message of any length into a message of fixed length. Where it is infeasible to derive the message from the hash fixed length message.

-Alice wants to send a (m)message to Bob.

-Alice and Bob use a symmetric cipher with a key shared some other way.

-Alice then generates two truly random numbers of same length as the message (m) and performs a OTP on the message twice, one key applied to m and one key applied to the result of the other key and m.. Alice destroys the keys.

-There is a server called a "hash server" which is a 3rd party member trusted by Alice and Bob and all other message senders.

-Alice send the double OTP ciphertext to the server.

-The server then replies with a fixed length message shorter than the cipher text and calls it the hash of (m) which the server doesn't know. This fixed length message is a truly random number which is stored on the server and sent to Alice.

-Alice concatenates the double encrypted OTP with m and uses the symmetric cipher to encrypt that.

-Alice sends the (MAC) along with the ciphertext

-Bob sends the double encrypted ciphertext to the server. The server checks if that matches anything in its database and sends the resulting "hash" to Bob. If the result matches Alice's MAC then the message is intact.

There are no collisions since the server will never send the same random number out for two different messages. The double encryption is so that Bob cannot derive Alice's OTP key from the message he got from her.

Answer 1

I am kinda confused about what you are trying to achieve with this rather complicated and error-prone protocol.

If I understand your idea correctly, you are basically just moving the problem of integrity onto the communications with the "hash server". If someone was able to MITM both Alice's and Bob's in- and outbound traffic, they could mess with Alice's message, then make the server send an arbitrary "hash" to Alice when she requests one and return that same value to Bob when he asks for a "hash" the next time.

There are also some other assumptions that seem contradictory for me; For example, it is mathematically impossible for the server to "never send the same random number for two different messages" if you also require that every hash is a "fixed length message shorter than the cipher text". Also, I see no reason why Bob shouldn't be allowed to "derive Alice's OTP key" - what would he do with it anyway? Thus, the "double OTP" encryption seems unnecessary to me.

You have probably heard it before, but there is this rule in cryptography that says "don't roll your own", which basically means you should not try to invent your own protocol if there are established solutions that haven't been broken so far. The reason is that these well-established protocols have usually undergone a lot of research from large numbers of very intelligent people, as opposed to your "new" algorithm that you have put on the stackexchange forums for discussion ;)