The "keyed MD5" described in RFC 1828 can be summarized as follows: for key K and data D, the MAC value is MD5(K||D||K) (in the RFC, K is first padded to a length multiple of 512 bits, but it does not substantially change things here). To my knowledge, there is no known weakness to that construction, but there is not much security analysis either.
HMAC has been designed to provide security when used with a hash function that relies on the Merkle–Damgård construction. It has been proven that IF the inner "compression function" of the hash function acts like a pseudorandom function family, then HMAC is secure. We do not know of any similar security proof for the "keyed hash" construction. In that sense, we believe more in the security of HMAC than in the security of the "keyed hash".
If hash functions were "perfect" in a sense that relates to the notion of random oracles, then the keyed hash would be "obviously" secure, like a number of other constructions, e.g. h(K||D). This latter example falls to the length extension attack when the hash function uses the MD construction (which is the case for MD5 but also SHA-1, SHA-256...); this does not directly impact the "keyed hash" from RFC 1828, but is sufficient to demonstrate that MD-based hash functions are not perfect in the sense alluded to above, and thus security of any MAC construction cannot be simply assumed. We need some more details, and with HMAC we have these details.
Some extra notes:
The "length extension attack" does not contradict the classical security of the hash function, namely resistance to preimages, second preimages and collisions. In particular, SHA-256 and SHA-512 are "believed secure" and yet fall under the scope of the length extension attack.
Collisions are not an issue here, and, more generally, are not an issue in constructions that use inputs unknown to the attacker (as is customary for MAC, because of the key). We know how to produce collisions for MD5 very efficiently; but we do not know how to attack HMAC/MD5 or even the "keyed MD5". (It is possible to design a contrived MAC that uses MD5 and is weak because of MD5 collisions, but that's not the case for HMAC nor for the "keyed MD5" we are discussing here.)
Conversely, while collisions do not apply to HMAC, the mere existence of efficient methods to compute MD5 collisions demonstrates that the inner compression function of MD5 is not a PRF, and thus the HMAC security proof does not apply to HMAC/MD5. This is a pity since the whole point of using HMAC and not the "keyed hash" is to benefit from the security proof. Take care to note, though, that lack of security proof does not mean that attacks are possible; only that if an attack is found, then it would not contradict the generic HMAC security proof.
