The reason why you use a KDF, or a secure hash for that matter, on a curve25519 shared key is that the bits are not distributed randomly. You have 32 bytes of "point data" which contain roughly 126 bits of "security".
So... which bits do you choose? Take the first 126 bits and leave the remaining 2 bits of your 128-bit key zero? Or take the last 126 bits? Or just take 128 bits out of the middle? Some other strategy? How do you know you picked the right bits? How do you know there are no exploitable patterns?
Using a secure hash or KDF solves all these issues. Something-something-input gives 128-bits of pretty-much-perfectly-random output (or rather, random-looking). Or, any other amount of bits that you want. You do not waste entropy, you need not worry whether you picked the "good" bits, and you do not risk having possibly exploitable, obvious patterns that come from the ECC's calculations (which are not perfectly "random looking"). Of course, entropy will not "magically" be added if you stretch, but the point is, an external observer cannot tell where it is. The KDF or hash does not need to be slow (and most of the time should not be).
The reason why you use a slow hash or KDF on passwords or any other user input is that anything that comes from a human has embarrassingly poor entropy, and is subject to being brute-forced with the aid of a dictionary (plus obvious permutations). Modern computers can literally do hundreds of millions of simple hashes per second, so that's a problem if your password database gets stolen. The attacker may not break the complete database, but getting a few users' passwords is a matter of a fraction of a second if no deliberately slow function is used. The longer it takes a prospective attacker, the better. More work needed to break a password means you have more of a time window to react and inform users in case of a breach.
The same goes for e.g. access to your encrypted disk or your Keepass file. If an attacker can try 100-200 million passwords per second, you may as well not encrypt at all, it doesn't matter how much care you put into choosing a good password.
If an attacker can try 3-4 passwords per second because that's just how long it takes to run the KDF, your password is basically "unbreakable" because it takes forever to find a match at that rate.
Certainly, this also makes it more expensive for you to unlock your volume. However, you only do it once, an attacker must do it many times.