My main concern is not being a pain in the ass to my end-users. However, because 'banks' do it, some believe that session time-outs make an app more secure. But does it when your only mode of authenticating is username/password? (given that session keys are properly randomized and user interacts only over HTTPS?). I tend to disagree, but would love to be proven wrong. I have a customer complaining that my site is insecure because I don't time out sessions within half an hour.
Session timeout can prevent or mitigate several types of attacks. The following come to mind.
Unsecured workstation. Boss leaves his desk for lunch, employee sits down and gives himself a raise.
Shared devices. Donald Trump uses a complimentary iPad at the Mar-a-la-go Resort. Terrorist is able to obtain iPad afterward. Obtains nuclear codes.
Brute force. FREAK attack downgrades SSL/TLS encryption to 512-bit; server farm containing 1,000 PS-4 gamestations cracks the key in just over 31 minutes. Only those sites with a 30-minute session timeout escape catastrophe.
Session identifier theft. Odd Job Trojan infects browsers all over the world and sends session keys to one hacker located in Hoboken, New Jersey. Hacker has all the time in the world to sift through the harvested keys and use them for nefarious purposes. Some of the keys will no longer work by the time he gets to them-- those with session timeouts.
CSRF. Hacker sends out a million emails with a funny GIF that links to
https://www.wellsfargo.com/Transfer.aspx?ToAccount=1234&Amount=100000. Of the one million email recipients, 10,000 open them. Of those, 100 are still logged into Wellsfargo because their session timeout is too long. Hacker gets millions of $$$, retires in Costa Rica, where he now passes time by posting on Stack Exchange.
If you don't implement session timeout, you are more vulnerable to all of the above sorts of attacks. Using an HTTPS connection and/or using a crytopgraphically random session identifier are good and important steps, but neither of those saved anyone in the attacks above.
Only you can decide if the risk of the above attacks is important enough to offset the additional inconvenience to the end user. If it's just a social app, maybe not. If it's a banking app, almost certainly yes.
There's a trade-off between ease of use and security. You'll need to make appropriate choices based both on how your users interact with your site and your threat assessment of what would happen if an unauthorized user got access to their account (from being left signed in after many minutes of being inactive).
For example, if the typical user want to stay logged in for several hours with long periods of inactivity interspersed in between -- say with an webmail program, group chat application (e.g., IRC, slack), social media platform (e.g., facebook/instagram/twitter), then it would not make sense to force them to logout after five minutes of inactivity. Forced logouts requiring repeatedly signing in would frustrate users who would migrate to other platforms.
On the flip side, if you are a bank and users typically sign in to do something quick (e.g., check a balance, pay a bill, or transfer money) and rarely stay in for long periods of time, it makes perfect sense to log them out after a few minutes of inactivity.
It also makes sense to think about worst case scenarios. If someone leaves their house while still signed into their bank account on their computer, a thief breaking into their house could potentially transfer their life savings to an account they control. Meanwhile, if someone leaves their house while still signed into their social media account, they may be slightly embarrassed (and have to explain their account was hacked).
There may also be scenarios where the threat of unauthorized use is so great that every time that action is performed it needs to be authenticated. E.g., before you change your password, you always require the user to re-authenticate. Or before you checkout your cart at an e-store (especially for digital goods that can't be returned) you maybe always force the user to authenticate.
Banks use 2-factor authentication. That second factor, a token or whatever, is unlike a pasword not something that a user can permanently store in his/her password manager. Hence, requiring a user, after a certain time of inactivity, to re-identify.
When using just username/password authentication it makes a lot less sense, since typically these days browsers or external tools store basic user credentials. Bothering users with a screen that just requires them to submit a pre-filled form doesn't increase security. It's nothing but bothering your end-users.
It is, however, important that session IDs cannot be brute-forced. Unless you're building your own authentication scheme, modern day frameworks will probably have randomized session IDs that cannot be guessed within weeks of brute forcing.