I have a memory dump. I know it is infected by a rootkit which uses its own file system installed at the end of the disk. Can there be any artifacts in dmp file to show this?
The answer all depends on the rootkit and how it operated. There WILL BE remnants, but whether they are visible, encrypted, mangled, etc., are another story. Since you have a memory image, you can analyze it with Volatility, carve out data from the dmp with psdispscan if you can detect the anomaly from the memory. You could also use scalpel to do this but again, you need something to base your carvings on.
Had I to perform these tasks, I'd use a combination of Volatility, Memoryze, and Redline. I would try to carve out as many processes as possible, any anomalies, and perhaps shove some of them through Virustotal or Anubis. It all depends on my objective(s). So the short answer, yes there will be remnants, the conclusive answer: There are remnants you may not be able to see.
You can hash/extract each artifact (in a VM) and compare that with a malware database. I assume you have already taken a forensically sound image of the system of course.