I have a heterogeneous system (both MS and *nix) that communicates with CIFS/SMB. How can I ensure proper data encryption at the application layer?
If it is at the "application layer" then it is done by applications, on what applications see, i.e. files. In other words, the applications shall choose a common format for encrypted files. Consider OpenPGP format as a starting point, and GnuPG as an opensource library and command-line utility which knows this format. You still have to decide what kind of security model you wish to enforce; a simple common key shared between the relevant installed applications might be enough, or not, depending on what you are trying to do (encryption is like medecine, it is not a single thing that can be generically applied everywhere; there are details).
If, on the other hand, you would like the applications themselves to be blissfully unaware of any encryption, such that the applications only see a "normal" CIFS/SMB which gets encrypted under the hood, then, by definition, this is not encryption "at the application layer" but at some other, deeper layer. I suggest using a VPN. IPsec is a security protocol which adds VPN-like features to IP, and which many operating systems implement (including Windows, and the non-prehistoric Unix-like). A VPN will be good if your intended security model is about attackers who spy on the communication lines between the involved machines; it will not help in the case of shared files if the machine on which the files are physically hosted is also potentially hostile.
Use Samba 3.3.x+ and set
server signing = [auto|mandatory|disabled] in the Global section (disabled by default). Share level SMB encryption is auto by default. This has been tested with WinXP/Win7 and AIX 5.3 running Samba 3.6.7. SMB encryption became available in Samba 3.2 but server signing did not appear until 3.3. These are required for Win7 clients configured to Microsoft's security recommendations (NTLMv2 and 128 bit encryption).
Although you may have figured this out, I was looking for the same information and might have found something a bit more helpful. What you appear to be looking for is "encrypted network transport", and it is available in Samba since version 3.2 (at least).
Sadly, it is bloody hard to find out how to set this up by looking at the docs (even the Samba Wiki does not appear to have info about it), but one way to do it is by using the
-e option for
smbclient. You may have better luck finding details about how to mount a Samba share using encryption, but I have been unable so far (I imagine it has more to do with server configuration than client configuration).
This page about SMB transport encryption in Windows may also be helpful.