The GUID's in the proj and solution files are generated when you create the project and solution files, so copying the source and recompiling on another machine will probably not change the GUID's (I haven't tried it though but i guess they'll only change when moving things around in Visual Studio).
Besides, I don't think those GUIDs will end up in the final executable, you should be more concerned about the GUIDs defined in the source, mainly in Properties/AssembyInfo.cs (they may be defined elsewhere, but that's where Visual Studio puts them by default). These GUIDs do not change if you don't explicitly replace them, so if you build an open-source project, it's not possible to trace it down to your computer by, for example the assembly typelib GUID.
That said, who knows what the .Net compiler is adding as extra meta-data? It's possible that it generate GUIDS for version comparison between the PDB files and the executable for example (however I guess a hash is more likely)? The best way to test this is to build the same project on two machines and then compare the binarys.
Off topic: not to be inquisitive, but I am quite curious about the concern?