First, signing and verifying are NOT 'encrypting' and 'decrypting' the hash. That harmful trope is almost partly true for RSA but not really, and it is 100% totally wrong for DSA and ECDSA. See
Trying to understand RSA and it's terminology?
Public key authentication: what gets signed?
Is encrypting data with a private key dangerous?
If the public key can't be used for decrypting something encrypted by the private key, then how do digital signatures work?
Understanding digitial certifications
Digital Signature and Verification?
With GPG, can you "decrypt" a file that hasn’t been encrypted?
http://crypto.stackexchange.com/questions/15997/is-rsa-encryption-the-same-as-signature-generationhttps://crypto.stackexchange.com/questions/15997/is-rsa-encryption-the-same-as-signature-generation
First, signing and verifying are NOT 'encrypting' and 'decrypting' the hash. That harmful trope is almost partly true for RSA but not really, and it is 100% totally wrong for DSA and ECDSA. See
Trying to understand RSA and it's terminology?
Public key authentication: what gets signed?
Is encrypting data with a private key dangerous?
If the public key can't be used for decrypting something encrypted by the private key, then how do digital signatures work?
Understanding digitial certifications
Digital Signature and Verification?
With GPG, can you "decrypt" a file that hasn’t been encrypted?
http://crypto.stackexchange.com/questions/15997/is-rsa-encryption-the-same-as-signature-generation
Also, the signature algorithm field in the certificate is irrelevant; that's the algorithm for the signature of the certificate by the CA, and not the signature on the data by the EE (here the server) using the certified key.
Those said, what is signed is the two random values, plus the encoded form of server-params.
Here's an example. I captured a (test) handshake in Wireshark and here I export the ServerKeyExchange message to temp3.raw:
I also exported the client and server hello nonces to temp1.raw and temp2.raw respectively, and the subjectPublicKeyInfo from the cert to tempk.raw and converted to PEM form (which OpenSSL prefers) with openssl pkey -pubin -inform der <tempk.raw >tempk.pem. Now:
$ xxd temp1.raw
0000000: ead0 1445 aec2 291d e316 8f77 b75c e0ec ...E..)....w.\..
0000010: 696c ced0 9d45 65b3 78bd d6f8 5e11 287e il...Ee.x...^.(~
$ xxd temp2.raw
0000000: 5834 4f06 be02 498d cba5 16b0 80bd d5f3 X4O...I.........
0000010: 4a0c 1476 d085 8c57 6fbf 5485 47cd 3937 J..v...Wo.T.G.97
$ xxd temp3.raw
0000000: 0c00 0091 0300 1741 04da 0889 2ed6 bdd0 .......A........
0000010: d140 30e8 417a 7d15 9d46 55b5 6eed 4fde [email protected]}..FU.n.O.
0000020: 814c a7aa fd1d 6617 2835 75c4 12d4 a2a4 .L....f.(5u.....
0000030: 213e 6141 112c fb8d 620b 21f9 e27f ea54 !>aA.,..b.!....T
0000040: 53ec 9ec1 bbbc b3df 2f06 0300 4830 4602 S......./...H0F.
0000050: 2100 fd47 429f 478b 3bd4 c8d4 ce4a 9e4c !..GB.G.;....J.L
0000060: f6e0 c957 bf83 017d 0812 bb4a 8fda de06 ...W...}...J....
0000070: ddc0 0221 00e3 05b1 3046 3c32 a387 012f ...!....0F<2.../
0000080: d149 e810 b423 b45c cce1 e4e9 62ce bd6a .I...#.\....b..j
0000090: a4a0 b6d2 a3 .....
This ServerKeyExchange consists of the handshake message header (4 bytes), the encoded curve_name and pubkey value here totalling 3+1+65=69 bytes, 2 bytes signature and hash identifiers (TLSv1.2 only, for earlier see the RFCs) here 06 03 meaning SHA512 ECDSA, 2 bytes length, and 72 bytes actual signature. I construct the input and separate the signature, and use OpenSSL commandline to (hash&)verify:
$ (cat temp[12].raw;dd if=temp3.raw bs=1 skip=4 count=69 status=none) >temp.dat
$ dd if=temp3.raw bs=1 skip=77 count=72 status=none >temp.sig
$ openssl sha512 <temp.dat -verify tempk.pem -signature temp.sig
Verified OK