Can I just log out by wiping cookies instead of hitting logout?
Frequently yes, for the reasons you supplied in your question: Without the session token in your cookies, a typical web application won't know who you are.
What are the issues of just wiping cookies versus clicking the logout button?
Web applications that manage authentication following the OWASP session management guidelines will invalidate the session on the server side when you log out explicitly. If you simply discard the cookie with the session token, the session may be subject to session hijacking.
Using door locks as an analogy for those who are not familiar with best practices of developing web applications (thanks to discussion in the comments):
Your account can be seen as a room in a building. When you log in, the building's owner creates a door and puts an automatic lock on it, so that only you can enter. Your session token is your key, and is typically stored in your browser's cookies, but can be stored in other places.
Discarding your token by deleting your cookies, clearing cache, etc., is simply destroying your copy of the key.
Explicitly logging off is asking the building owner to brick up the doorway. There's nothing guaranteeing that they'll secure your account, but as the user, you're explicitly making your wishes known.
There are various ways that an attacker can get a copy of your key, known as session hijacking, that are the responsibility of the site owner to mitigate, not the users.
First, the attacker can just guess. If the site generates session keys sequentially, or uses a low entropy pseudorandom generation method, this makes guessing much easier. Sites mitigate this through using high entropy tokens and periodic session recycling. Recycling sessions doesn't prevent access, but it makes it obvious when unauthorized access has been granted.
Second, the attacker can use session fixation: They give you a key before you log in, which you continue to use after you've logged in. Sites mitigate this by explicitly recycling the session when you log in.
Third, a Man-in-the-Middle attack. The attacker can see your key directly. TLS mitigates this. It is possible to decrypt TLS traffic through downgrade attacks, insecure implementations, and zero-day attacks, but these are far outside a user's domain, rare, and zero-day attacks against TLS tend to raise a LOT of noise when they're discovered (Heartbleed, et al).
As a user, your responsibilities are to log off and to hold the site accountable when they take shortcuts with security, much as your responsibility with your car in a public parking lot is to lock your doors. If the door locks are trivially bypassed, then its the manufacturer's fault, not yours.
