When acting as an authorization server (AS) that handles OAuth 2.0, i think two major security features of the library you should check are:
Preventing access token leaking by restricting the redirect URL. In the implicit flow, the AS receives an authorization grant from a client it can not trust. In order not to give away access tokens to anybody, the library should allow you to specify a (white-)list of URLs for every connecting client. In the penetration test, you should check whether you can use a URL in the redirect_url that doesn't match one of the URLs in the whitelist (that would be a fail).
Preventing Cross-Site Request Forgery and/or Session Fixation with the state parameter. The state parameter is a random value, generated by the client. The parameter will then always be included in subsequent communication with the AS. It allows the client to validate that the access token it receives corresponds to the preceeding authorization request. Without this parameter, an attacker could inject his own access token, and therefore grant access to resources that the attacker controls. The end-user would then e.g. publish his data on the attacker's account. So for the penetration test, you should check whether the library properly validates the state parameter, or if you can alter the value.
Oh, and I almost forgot: OAuth 2.0 clearly specifies that it relies on secure SSL/TLS transport. So if you're not exclusively servicing your AS over HTTPS, that's a big #FAIL in the penetration test report.
The initial OAuth 2.0 specification RFC 6749, section 10 already contains lots of security considerations. Moreover, there's an additional RFC dedicated to OAuth 2.0 security: https://tools.ietf.org/html/rfc6819
Another great resource that sums everything up can be found here: http://www.oauthsecurity.com/