Because the lead developer felt a new format was necessary for Ed25519 support within OpenSSH:
Ed25519 is not supported in OpenSSL, so we used a public-domain
implementation (from SUPERCOP). Unfortunately this means that we could
not use the PEM key format that we have used for RSA, DSA and ECDSA keys
until now, so Markus made a new one.
The new key format looks a lot like the old one (a blob of base64
encoded key material with beginning and end markers), but offers quite
a bit more protection to the key from offline attacks against the
passphrase. The new format uses a bcrypt-based key derivation function
that makes is brute-force attacks against stolen private keys far
So far, it is only required for Ed25519 keys but it is possible to
request it for other key types too by adding the '-o' flag to ssh-keygen
when generating a key. It's also possible to convert existing keys to
the new format by specifying the -o flag when changing the passphrase
The PEM handling code is nasty and OpenSSH relies on OpenSSL to do most of that heavy lifting. OpenSSH would have had to roll its own PEM code to support that, so the lead developer (Markus Friedl) decided to introduce a new format that not only was backwards compatible with PEM and supported Ed25519, but also came with some other features.
This is not without precedent: Putty implements the PPK format and is quite opinionated about it.
Here are a few links to the technical details of the new format: