So I'm asking about Linux or Windows type of rootkit/malware
is this possible? can a rootkit use the slack space in the pages of the memory instead of disk to hide data?
if so, has this ever been used in a malware? and how is it done?
Of course it can, the question is why would it do that. A rootkit is a a piece of software that injects code into the kernel, it might as well use that to disguise itself.
There are several issues with RAM slack:
You basically have to find the address of something that you know will have slack and write after the end of it.