As far as I know, with OpenSSL, you can self-sign your website's certificate. This means that the browsers that will connect your server are supposed to be willing to accept a self-signed certificate for your website. My question is, how does the browser know whether or not to accept a self-signed certificate for a particular website? What if I MITM a client and present a self-signed certificate for a bank website? How does the browser not get tricked at that point?
Browsers will only accept an invalid certificate - and self-signed is one form of "invalid" - when the user acknowledges the risk and overrides the browser. The specific steps for doing so vary from browser to browser, but they're usually onerous by design - they want the decision to bypass security to be hard, not easy.
Browsers will alert the user if they are presented with a self-signed certificate which they don't trust.
The browser user or system administrator should preempts this scenario and add the self-signed certificate to the browser's trust-anchor store beforehand. That way, the user won't see a warning.
Done this way, users can be informed that if they ever see a browser warning they should stop browsing and report it.
Instructing users to simply bypass the security alert shown when an unknown self-signed certificate is presented by a server means that in the event of a real MITM attack the users will do exactly as the attacker wants them to do and simply continue to the site.
By root certificate which is included in each browsers already. There are lots of root certificate in browsers, and it is referred in every TLS or SSL session using chain of trust. At last, the root certificate in TLS/SSL session is verified if the root certificate is installed in the system, browser.
If a user/admin insert the self-signed root certificate in the system or browser, it will work well.
It means that the list of root certificate is important for securing TLS/SSL. Hence, mozilla and android and most browser related organizations maintain the list of root certificate.
CTS(Compatibility Test Suite) for android enforce the list, it means that if any not permitted root certificate is included in factory phase, google cert. would be not issued. https://android.googlesource.com/platform/system/ca-certificates/+/android-cts-8.0_r5
What if I MITM a client and present a self-signed certificate for a bank website? How does the browser not get tricked at that point?
Hopefully the certificate would have name constraints that would prevent it (or any certificate signed by it) from applying to arbitrary domains. (Sadly this is not the typical case, so an attack of the type you describe would likely succeed if you could intercept the traffic.)