I would assume that A/V products would need to employ the same kinds of techniques used by the software they are trying to cleanse to ensure that they can continue to run in such hostile environments. However, A/V products are not exactly the most stealthy -- you can find evidence of them all over a Windows machine.
What are A/V products doing to protect their integrity and availability?
Any antivirus application worth its salt is going to be almost indistinguishable from malware if one only looks at its self-protection mechanism. If we wanted to be flippant, we could say that the only difference between a rogue security tool and a legitimate security tool is the tool designer's intent.
While AV's generally don't attempt to conceal their presence on a system (as rootkits do,) they will intercept any system call which the designers feel presents a threat to the application (like many, many modern examples of malware do.) Calls like TerminateProcess or CreateRemoteThread are obvious targets for a self protection scheme; another good target would be CreateFile since it's possibly the most used Windows API function of all (and can do much more than just create files.) An AV program can hook these calls and only allow them to proceed if they won't threaten the AV program. An AV may also install a Kernel mode driver so that it can manipulate the Kernel directly and interfere with malicious attempts to modify the kernel. Many of the self-defense mechanisms serve double-duty as part of the defenses provided against malware in general.
Ironic aside: some anti-malware programs are harder to remove than most malware.
Most of the time "integrity" is checked by self hashing and verifying that... in a best-case scenario with a safe server.
"availability" is a bit more problematic to answer, as different antivirs use different approaches.
One thing has to be remembered: although antivirs don't spread the news too often (for obvious reasons), both "integrity" and "availability" checks have a high potential of failing to detect failures if the machine is infected at a low level (like rootkits). Yet, antivirs already gained the option to check on (known and unknown) rootkits up to some extend.
Typically they try to understand if the environment is safe at the installation phase. Because if they overcome this phase you can assume the environment is safe.
At a lower lever, when they install kernel hooks to intercepts read/write, they check the syscall kernel table to see if other software have intercepted r/w OS syscall. It's the same technique that antirootkit software use.
I guess they scan the memory too from viruses and worms and check the integrity of their core files with a sort of fingerprint such as SHA1 or MD5.
But sure, if the machine is under a rootkit (in the enemy hands) they cannot do enough to protect themselves. They can survive enough to signal to the user that something is gone wrong, but not survive forever in hostile environment.
