I frequently hear that 600 and 700 is recommended for security when possible before 660, 770. I'd like to understand the risk that 660, 770 would have that 600, 700 wouldn't.
The principle at play here is the least privilege.
A chmod permission of 600 or 700 gives only the owner rights to the file, while 660 or 770 gives the same rights to the group too. Whether this is intended or not depends on the use case. Either might be appropriate.
A server might - and will probably - have different categories of users (students, guests, employees, administrators ...). Keeping privileges separated is important in this case.
The possible problem with 660 and 770 is that a sysadmin might inadvertently add an extraneous user to the group in question, hereby giving the group-crashing user the same rights as the owner.
It is worth noting that nowadays group permission in Linux are obsolete (they've been replaced by ACLs), and by default an user is assigned only his/her group.
The risk from "inadvertently add[ing] an extraneous user to the group" is minimal. And fACLs are not obsolete. Used properly fACLs supplement the the things which are difficult to do with the base permissions system, but they do open the possibility of creating a labrynthine mess the likes of which are only normally seen on badly managed MSWindows shares.
Unix is intended as a multi-user system - and has a permissions model that allows users to have their own files, files they share with a named group and files available to anyone with access to the system. Sadly, too many people don't understand how to use the second of these scenarios (and I include a lot of people paid to have expertise in Security and Unix in that category). But we don't really expect normal users to have a proper understanding of the distinction.
Shouldn't group users already be trusted?
While the 'user' and 'other' are well defined, the 'group' permissions are specific to the group ownership of the file. That means I can share a file with 'finance' without exposing it to 'webdevelopers' and if I create a hard link to that file I can also share it with 'marketing'. However when a user creates a file, then it will be owned by the users default group. If the permissions are 660 it will be accessible to the default group. OTOH if the umask is set to 0077, then the user has to take specific actions to share that file with others (although the same result can be achieved by creating a new group for each user and setting that as their default group).
There are lots more permissions than just 770/700, 660/600 (2770/2775 is one of my favourites) 600/700 is just a safe starting point until you explicitly tell the computer whom you want to share the file/directory with.