I don't know if this is the right answer, but after a bit of research, I came up with this 'solution'. First of all, ensure you have a secure boot chain, which can guarantee no firmware rootkit is installed on your device, up to kernel level. This ensure the integrity of the system is trusted when it is booted. So now we have to worry about a runtime malware happen after the boot chain. First of all, we need some application sandboxing to minimise the attack surface, and we know that if an attacker exploited another application that is running on the device, they will not be able to view / touch our application.
And let say, unfortunately, they were able to exploit the Operating System, we hope we can somehow detect some artifacts before they reach this point, and our server can treat this device as compromised / untrusted.
In MacOS, there is something called the SIP (https://en.wikipedia.org/wiki/System_Integrity_Protection), which should be able to protect the process even if they gain root access. Please correct me if this is not correct. I think the idea is we now move the trust to kernel, so unless the attacker gained kernel access, they should not be able to touch our logging process.
Well.. and let say they have a 0-day kernel exploitation, does it mean we have to surrender? This is another chance for us to catch them before they reach the kernel. They have to exploit some kernel API and run arbitrary code as kernel. There are some protection for kernel at runtime such as Linux Kernel Runtime Protection, Windows VMS, Android KNOX etc. What they do is basically keep track of another list of task list, and hook into the syscall. They do integrity check when they return from the syscall. I am not 100% sure how this is being done, but I guess this might be a good chance to catch them before our logging agent is compromised.
There are a lot of research going on in this topic, a group called trusted computing group is working on this problem. However, there are a lot of difficulties due to the design of the operating system. When they first came out, they are not designed with mandatory access control. A superuser is able to do anything on the system. They are slowly progressing from Windows Vista, integrity level on processes, selinux on linux etc...
I don't think my answer is perfect but thats my understanding after doing a bit of research. It will be great if someone can clarify the points that I have mentioned above