Two things that it means to me are:
it's a stream cipher, therefore not parallelizable. Although it's a stream cipher, it has a randomly accessible output stream (PDF), so it can be parallelized (unlike RC4, and stream ciphers in general). Poly1305 is also parallelizable
SSH already has a set of counter mode block ciphers which are parallelizable (RFC4344 Section 4) though they don't get you a performance gain out of the box. ChaCha20 (20 rounds) is a little faster than RC4, comparing cycles per byte, and is faster than AES256.
- it's an extra algorithm in the SSH negotiation
The second point needs some elaboration: there have been a number of extra key-exchange (EC), MAC (SHA-2) algorithms added recently, and now another cipher. This increase causes problems with older (including embedded/devices) SSH server implementations. In SSH the various algorithms are spelled out, rather than having short, fixed identifiers like TLS. The workaround is to chop the list down with explicit
KexAlgorithms in your ssh
config. There's also the potential for protocol enforcing firewalls or IPSs to misfire.
I believe one protocol difference between TLS and SSH is that the authentication, key-exchange, cipher and MAC are identified and negotiated independently, with TLS each publickey-kex-cipher-mac set is one of a defined suite. Poly1305 adds authentication, so ChaCha20-Poly1305 is (effectively) an authenticating stream cipher, no separate MAC is required by SSH for it, the benefit of this is explained in David Miller's article linked to in the question.