The crucial test of the security of a proposed cryptographic PRNG is the next-bit test. That means, given some of the output of the PRNG, it should be impossible for someone to determine the next bit of the output with a probability better than 1/2 (that is, better than just guessing).
In your case, if we've seen 512 bits of the output, we can guess the next bits with probability 1: they'll be the SHA-512 hash of the bits we've seen.
There are two common designs that use cryptographic hash functions that are secure, and they're outlined in NIST's SP 800-90A Rev 1. These are the HMAC DRBG and the Hash DRBG.
I'll refer you to the documentation for the complete implementation, but roughly, the Hash DRBG works like this:
- Derive a seed from the input, and set V to the seed. Derive a variable C from V preceded by a zero byte. Set the reseed counter to 1.
- To generate data, set D to V. Hash D and put that in the output. Then add one to D, and hash it again for the next block of output, continuing until you've produced the desired amount of output.
- To update the state, hash V preceded by a byte with value 3, and call it H. Set V to V + H + C + the reseed counter. Increment the reseed counter.
Note that in this approach, the attacker doesn't ever see D, so they cannot determine the next output data from the previous data. This is ultimately the "hash of a hash", but the difference is that D is secret, not something we've output to the attacker.
There is also the HMAC DRBG, which in my opinion is easier to implement, because it uses a hash function with HMAC and doesn't require bignum arithmetic, which is error prone and may not be constant time. It's described in the same document. It's also used in RFC 6979 as part of deterministic DSA and ECDSA, and is widely considered to be strong.
In general, though, you should use the system's CSPRNG:
RtlGenRandom, as appropriate. This will almost always be the right option and will be secure. Only if you are sure that you cannot use it or have other requirements should you implement one of these algorithms yourself.