If I protect my files with password and use rar or zip files to store these files. Can ransomware encrypt password protected files?
At the OS level, a file whatever is its content or structure is nothing more than a sequence of bytes. That means that the sequence of bytes can be read, encrypted and written again (in a crypted version) regardless of whether it was originally a text file, a JPEG image, or a ZIP archive.
So yes, password protected files will be encrypted by the ransomware. At at decryption time (if you pay) you will get back a copy of the original files that will still need the original password to be used, because the copy occured at the byte sequence level.
Can you put a password-protected zip file into another password-protected zip file? The answer here is yes, and you can try this out yourself. Ransomware does essentially the same thing, but this time you don't know the password to the outer file.
The reason this works is that encryption and decryption is just a way to turn one sequence of bytes into another sequence in a way that makes use of the password. Whether the input of this was produced by yet another encryption does not matter, since the meaning of the sequence of bytes is irrelevant for the process.