In general you don't need to protect the Public key on hardware storage. It's called the public key for a good reason.
If you encrypt something with a public key, then it is only possible to decrypt it with the private key (and vice versa).
So, in general it doesn't matter who has the public key. Your public key could be on a billboard and your server could encrypt with the public key as much as you like and nobody will be able to read the encrypted data except whoever has the private key.
All the PKI hardware (smartcards, HSMs, etc) I am aware of is divided into public sectors and private sectors so if you want to store the public key privately then the private key is loaded onto the public partition (i.e. not an ideal solution).
If you want something more like mutual authentication in the log distribution mechanism then it's much better to use two sets of keys (one set for the client and one set for the Server).