I got a document with integration with another application. They mentioned I need to hash and then encrypt data with my private key before sending it over HTTPS. By doing this, they are making sure that data was not altered in the middle of transit.
I wanted to know what is the reason for three layer security(SSL+SHA256+RSA). is SHA256 weak by itself or are there other concerns which I am not aware of?
The document your are reading uses the flawed analogy "encrypt with the private key". In asymmetric encryption, you encrypt with the public key, and you decrypt with the private key. "Encryption with the private key" is the way digital signatures were first described, because that kind of works like that in RSA. But the analogy does not work with other signature algorithms like DSA, and, when you look at the details, you see that it does not work with RSA either.
So just forget the word "encryption" here. They ask for a signature. A signature begins by hashing the data which is to be signed (with a hash function like SHA-256), and the rest of the computation involves only the hash output and the private key. The signature is not used in addition to SHA-256; SHA-256 is an element of the RSA signature process.
Why would they ask for a signature ? SSL already ensures data integrity. But they may want non-repudiation. When using SSL, client and server establish a tunnel for data; the client has some guarantee that it talks to the right server; if some authentication occurred (e.g. with a user name & password sent through the SSL tunnel), then the server also has some guarantee that it talks to the right client. The SSL mechanics also guarantee that the data is not eavesdropped on or altered while in transit.
What SSL does not do, however, is to produce proofs which could be shown to a third party (e.g. a judge). If there is some ulterior legal trouble between the client and the server, the server has nothing to show to a judge to demonstrate that the client sent a specific request. When the server receives the request, it is well convinced that it comes from the client, but the server cannot prove that it did not make a fake request to incriminate the client. To have such proofs, you need something else. That something else is partly technical, partly legal; the technical part is digital signatures (the legal half depends a lot on the jurisdiction, so there is no absolute answer here).
Summary: the other application wants a signature. SHA-256 is part of the signature process, not a redundant step. A signature is requested probably in order to achieve some sort of non-repudiation, which raw HTTPS does not provide (it might also be an overkill of an authentication system, though).
