I don't suggest that you use either. You need to develop your skills without the tools in order to know what to attack and how to attack it.
The following books will get you started:
- Network Security Assessment, 3rd Edition
- Cyber Operations: Building, Defending, and Attacking Modern Computer
- Python: Penetration Testing for Developers
- Mastering Kali Linux for Advanced Penetration Testing, 2nd Edition
- Metasploit Revealed: Secrets of the Expert Pentester
You must master the techniques without using the tools. Metasploit is a sort of technique-building framework, but you can also rely on it too-much if you don't learn the scripting or code-specific relevance to each module down to the details, not just the options or advanced options.
However, nmap can be modified to gain an understanding about an environment in terms of vulnerability, control efficiency, and deep security practice. I would recommend it over Nessus for every situation except for the Fire-and Forget situation. However, Fire-and Forget tends to provide a report with the wrong risks and a huge-variety of errors including false positives (which can be false-false positives if the expert driving does not know how to reduce false positives to the right problem sets) and false negatives.
Thus, I do also recommend (after the techniques are mastered) these books:
- Nmap: Network Exploration and Security Auditing Cookbook, 2nd Edition
- Mastering the Nmap Scripting Engine