The point of using the key file in KeePass is that you can keep the file out of reach (on external media) and connect it only when you need to unlock the database.
As you consider phones, you likely have no way of moving the key file outside of the regular filesystem. That in turn means it does not bring any security benefit. Refer to this guide, especially the "Location" section.
If you use composite method: password and key file, having the file on your phone nullifies the value added, so your assumption "I use a key file [ ] I'm safe from dictionary attacks on my master password" is false.
In effect decreasing the number of rounds for the password has the same effect regardless if you use the key file or not.
If you use only the key file, decreasing the number of rounds does not affect you security, as it is already zero.
Technically, according to the documentation, the key file is not subject to the multiple key transformation rounds, instead its SHA-256 is calculated only once:
When using both password and key file, the final key is derived as follows:
SHA-256(SHA-256(password), key file contents)
To generate the final 256-bit key that is used for the block cipher, KeePass first hashes the user's password using SHA-256, encrypts the result N times using the Advanced Encryption Standard (AES) algorithm (called key transformation rounds from on now), and then hashes it again using SHA-256.