As someone who has actually done this for a couple of years, I can tell you that it's not nearly as simple as you're describing and it doesn't offer the security properties you want.
In summary:
Why do so many security experts (i.e. cryptologists & co.) not host their own email?
Because it takes enormous amounts of time and specialized knowledge to do it properly, it's not much more secure in general (it's actually less secure in some circumstances) and there are better and easier solutions readily available
How much time do you need?
Your initial time estimates are very optimistic, even assuming you're a knowledgeable system administrator with lots of experience on the task.
To give you a quick idea, a popular book that covers one of the most popular MTAs spans around 500 pages.
That's just for the MTA. In the real world you'll also need an MDA with POP/IMAP, spam filtering and probably a authentication server.
You're also neglecting the continued effort needed to keep the system healthy and secure:
Monitoring machine and service health (are there OS updates? Is the service running and responding? Are you running out of disk space?)
Monitoring logs, diagnosing errors (are there ongoing DoS attacks? is someone brute-forcing SMTP auth? Why are logins not working?)
Security-related monitoring (Did tripwire
alert you because of a OS update or a intrusion? Were there any security advisories of your OS this week? What about all the other software running on the machine?)
Unless you're happy having your messages severely delayed occasionally (or lost, in the worst case), there's a lot of other things that are also needed to reliably provide email (backup MX, failover and storage redundancy come to mind). A single home server probably won't be enough if your connection drops and you need to respond to a email urgently.
How secure can it be in the best case?
You didn't specify your thread model very clearly, but it seems you're concerned with a state-sponsored actor that is interested in accessing your email in particular. The setup you've described will not prevent that. As an example, there's strong evidence that the Heartbleed bug has been extensively used before its public discovery. If you were a sufficiently interesting target running a email server, compromising it would have been no problem at all. A sufficiently well-funded adversary will have the capability of compromising any practical security system you can devise.
Self-hosting also has the severe disadvantage of exposing much more information about you, if your system has few users. Your ISP can trivially tell when you receive or send an email, and with which providers you communicate.
Of course, that's not to say the self-hosted model is useless against other threat models. It prevents disclosure of your email to your service provider and helps you stay secure were a massive security breach to occur, since it's a isolated system. It also prevents silent coercion (lawful or not) of the provider. Some of these problems can be partially mitigated by using a competent provider in a country with strong data protection laws.
Are there better alternatives?
If you need security with this threat-model in mind, there are better and easier solutions, as Snowden has said himself:
Properly implemented strong crypto systems are one of the few things that you can rely on.
PGP is unrivaled against this particular threat model, as you don't need to trust any server or provider at all.
Compared to maintaining a email server, PGP is actually quite easy to use and understand.