When debugging some C code, especially tracking down after-free-accesses bugs, address space randomization is quite inconvenient, because it makes bugs non deterministic. By turning it off, you can much more easily reproduce the issues. That's a good reason to turn it off on development machines (as opposed to production systems).
As for the slowing down, there is a theoretical effect due to the spreading of "interesting addresses" over a larger space. Since the MMU on x86 hardware uses forward tables to represent the address space (a mapping from virtual space to physical space, contrary to the reverse tables used in recent PowerPC processors), it can be predicted that address space randomization will induce slight additional pressure on the caches (more cache memory will be used to read parts of the MMU tables). However, as for all performance-related issues, I strongly suggest that such a slowdown should be duly measured and confirmed before taking harsh measures. In other words, turning address space randomization off is worth a try if you are desperate for performance on a given system, but it should not matter much on average.