Oh man, you're in for a treat this evening!
Is it possible for other application running in the OS to read the keystrokes that my application uses?
What about mouse events and window handles?
Absolutely. Windows message processing is really complicated. It works a bit like this:
LRESULT CALLBACK WndProc( HWND hwnd, UINT msg, WPARAM wParam, LPARAM lParam )
switch ( msg )
and so on. Basically, this function is continually called for each message the graphics system has for your app. Each different type of message is identified by
hwnd gives you your window handle. The information for each type of command is stored in
lparam or some combination thereof.
It turns out there are a huge number of API Hooking Techniques in Windows. The good news is that one of them is
SetWindowsHookEx() which allows you to insert your own handler into another application's message pipeline and, well, steal their messages. That's everything - every button pressed, every menu, every resize, create, destroy, mouse move, redraw, keydown, keyup etc that happens on that Window.
There is another way to invite yourself in - use DLL Injection to get your code into the process' address space and then modify the running code.
None of this is easy, but it is certainly possible
Now, you wanted to know about defences. Well, UIPI, or User Interface Privilege Isolation, is one such defence. The privilege level of the process (or Mandatory Integrity Level) controls what other processes can do as that process. Vista uses this trick to isolate certain applications from each other. Next up; Window Stations, Desktops and Sessions. Did you know that when you get that UAC dialog, you're actually running on a separate desktop and
consent.exe takes a screenshot of your desktop and pastes it as the background for the new desktop?
Go back a bit. You can have multiple desktops per Window station and, guess what? Well:
Windows inside the Windows OS are children of Desktop objects.
So that means if you create a separate desktop for your application to run in and crucially handles to objects in the main desktop don't show up here. Windows uses this to protect the logon, lock screens, UAC screens. It's also used by me in a product I'm developing, and it's used by KeePass.
These features are fundamental to Vista/Windows 7 and they're unlikely to go away. To make the most of them, leave UAC turned on and possibly read this.
Now, Mac OS X. Disclaimer: I know very little about the OS X stack; however, I am familiar with Linux/X Server. Linux is increasingly moving towards RBAC/MAC-type systems with the introduction of posix capabilities rather than setuid bits and the like. However, in implementing SELinux in X, the developers ran up against a brick wall. X basically runs its server (the desktop) as root (should be sending you warning signals) and desktop applications (clients) connect to it locally and pass in nothing more than a cookie. See X authorization.
Fundamentally, X operates over some kind of transport, making your actions at least sniffable to outsiders if encryption is not used. Firstly, the Linux kernel might just give you permission to capture keystroke with
ioperm(). Failing that, if you have the magic cookie, you can use it to connect to the X server and run all sorts of commands. This requires a dodgy setup, of course, but it is feasible.
In terms of adding in isolation between clients and servers, XACE is the SELinux equivalent.
Now, how does this apply to Mac OS X? Well, not a lot - apparently, X11 is deprecated or something. Anyway, internally, Mac OS X uses a rendering engine of its own and its own Windowing code (Cocoa). Well, it turns out with some digging you can intercept messages on Mac OS X too. Cocoa is the desktop API; underneath it uses something called Quartz which also supports taps (source).
I am not fully versed on how this all works - however, I'm imagining it comes down to getting the appropriate privileges and just making the right calls. Mac OS X uses Unix Shared objects, so its callback mechanism will basically use function pointers. I'm not sure what privilege restrictions there are.
If anyone knows more about OS X, do feel free to take what I've said and expand on it in your own answer :)