Say you have a windows server that works as a PDC AD box. Is this a potential Attack Vector for a small 20 people buisness? Wouldn't it be better to simply have an encrypted database file with all the user info always with me? For example someone could enter the building and physically harm the server or gain access to the AD database Info.
Basically, Microsoft Active Directory is a database containing encrypted user information.
But anyway, regardless of the format you're using to store this information, someone able to physically enter the server room and steal / harm the server will damage this information and cause business disruption.
MS Active Directory is a robust technology used by many companies, and it will provide an acceptable level of security when it comes to user management, identification, authentication etc. providing it is configured properly and follows the Active Directory Best Practices.
Note that other technologies might do the same job and will provide the same level of security if they are configured properly as well.