I have two online accounts from different banks, one bank expires my session faster than the other, it's quite annoying.
I was wondering how long (or short) should a session be for it to be considered safe enough but not annoying?
Generic answer: ideally, the session should be exactly as short as possible, the limit being, indeed, the user's annoyance.
You make sessions because the user would not accept to reenter his password for every single page request. A session is a deliberate, controlled weakening of your security model; you accept it because otherwise the site would be unusable. However, your interest, on the server, is to keep sessions short.
Now, of course, this is a trade-off. User's annoyance is not binary. A bank will balance their own feeling of security against their perception of the user's annoyance (and the importance thereof -- the bank knows that it can thoroughly abuse the user, since bank account owners are somewhat "captive"). Apparently, your two banks chose distinct compromises.
The following applies to sensitive sites (banks, government services, etc.)