If I protect my files with password and use rar or zip files to store these files. Can ransomware encrypt password protected files?
Yes. The zip or rar file will be encrypted like all other files so that they are useless for you. The only protection is making backups to another physical storage.
This will help independantly of whether the files are in a password protected rar or zip format.
At the OS level, a file whatever is its content or structure is nothing more than a sequence of bytes. That means that the sequence of bytes can be read, encrypted and written again (in a crypted version) regardless of whether it was originally a text file, a JPEG image, or a ZIP archive.
So yes, password protected files will be encrypted by the ransomware. At at decryption time (if you pay) you will get back a copy of the original files that will still need the original password to be used, because the copy occured at the byte sequence level.
Think of your zip as a house (with your data inside the house). Your house has a locked door. The ransomware builds a house around your house with its own lock. You cannot get inside your house unless you can get inside the ransomware's house.
Can you put a password-protected zip file into another password-protected zip file? The answer here is yes, and you can try this out yourself. Ransomware does essentially the same thing, but this time you don't know the password to the outer file.
The reason this works is that encryption and decryption is just a way to turn one sequence of bytes into another sequence in a way that makes use of the password. Whether the input of this was produced by yet another encryption does not matter, since the meaning of the sequence of bytes is irrelevant for the process.