With a single infoleak and access to the binary you can calculate the other addresses. Is this still possible when you don't have access to the binary?
Possibly. It would depend on the leak and the type of address leaked. For example, an address for the heap, stack, or executable segment of the binary are of little use for conventional exploitation without their context and knowing the offset from the base.
However, it may be possible to determine the type of address just by looking at it, and some useful information can still be obtained; especially return addresses on the stack. If it appears to be an address from the binary's executable page, that may give you an idea of the addresses used for code in the binary. If it's also possible to exploit a buffer overflow, an attack with something like BROP may be possible.
If a pointer to libc is leaked, it may be possible to determine the version of libc and what the pointer points to (IIRC, the lowest n bits of the address aren't affected by ASLR and can uniquely identify the libc function), possibly with some educated guessing and by searching the libc database. Once offsets are calculated, this combined with a buffer overflow could allow you to perform a ret2libc attack to call any function in libc and gain code execution.