I think a lot of the problem with capability-based systems and our understanding of them is that there are few in the wild. However, you're in luck - FreeBSD have just added capsicum support to their 9.0 release. Capsicum is a research project to implement a [lightweight practical set of capabilities]. This work involved people at the Google Security Research team and includes an experimental chrome browser build designed to use the system.
What might be most of interest to you are the list of features:
- capabilities - refined file descriptors with fine-grained rights
Practically, will operate like SELinux, with obvious administration differences. Has analogies to type labels.
- process descriptors - capability-centric process ID replacement
Has analogies to SELinux domains. From a user perspective, may appear to be no different.
Now, in the next set of features you see why capabilities aren't yet in common usage:
- anonymous shared memory objects - an extension to the POSIX shared memory API to support anonymous swap objects associated with file descriptors (capabilities)
- rtld-elf-cap - modified ELF run-time linker to construct sandboxed applications
- libcapsicum - library to create and use capabilities and sandboxed components
Whereas SELinux requires kernel support and a policy and is ok to go from there, configuration problems aside, practically implementing capabilities requires a modified approach to the APIs available and subsequent preparation of applications to access them.
Now specifically to your questions:
Can they both prevent access to the same granular level?
Roughly comparable, yes. SELinux can, for example, describe an application domain and give it access to a certain folder for reads only. Capability based systems would, when initialising the process, simply not hand it a capability to write to that path, or refuse such requests subsequently. It's the how they're implemented that varies.
Are different attacks possible on SELinux in targeted mode than on a capability based operating system?
Honestly, it's quite hard to say. I don't know of any mass-deployed capability based operating system that's attracted sufficient interest to be seriously road tested. I expect, however, ultimately the goal will remain the same - persuade a process to execute code for you using its higher privileges (capabilities). They may vary to exploit the how in the same way buffer overflows vary depending on the program being targeted, but ultimately the fundamental goal is still the same.
You could always give it a go. There's also documentation on the API modifications I've talked about.
Disclaimer: I am not affiliated with the Capsicum Project, the University of Cambridge, or the FreeBSD Project in any way.