One of the basic things that a computer needs to keep track of is where in memory the instructions that are being executed are located.
This is normally done by the CPU using what is often known as the instruction pointer or program counter register. The exact name depends on the architecture, but the concept generalizes across all architectures. It's called the instruction pointer on Intel CPUs, and that's the term I'll be using in this answer. The instruction pointer is continually incremented while instructions are executed from memory, and also updated whenever a jump instruction of some kind is executed. Modern CPU architectures with multiple parallel execution pipelines and multiple cores makes what is actually going on inside the CPU more complex, but to an external observer, the simple explanation is sufficient to understand what is going on and the purpose of the relevant registers.
By virtue of the old Intel 8086/8088, 80186 and 80286 being 16-bit processors, the instruction pointer was normally expressed as a pair of 16-bit values known as the code segment (code selector in the 80286, but except in protected mode selectors function very similarly to segments) and instruction pointer. In assembler parlace,
CS:IP where the colon indicates the pairing of a segment and an offset as a pair to form a memory address. CS and IP here are CPU registers, tiny memory locations within the CPU chip itself, in this particular case each 16 bits in size. (You also had things like
DS:DX which was conventionally the location to where to read or write data in memory; DS was data segment and DX is a general-purpose register that in this case held the data segment offset. There were originally four registers intended for general use, referred to as AX, BX, CX and DX respectively, each of which could be addressed in terms of the high half -- AH, for example -- or the low half -- AL, plus a small number of general-purpose registers that had specific intended uses. Modern CPUs have many, many more registers, with dozens of registers being common and over a hundred not unheard of.)
In making the 80386 CPU in the late 1980s, Intel made a lot of changes, including extending the address space to 32 bits while keeping selectors as a concept. For this reason, the old 16-bit values were no longer sufficient, and the instruction pointer was extended to 32 bits. In order to maintain full backward compatibility (at this point, Intel had probably learned the lesson of the 80286 which was more one-way compatible; it was easy to go from the 8086/8088's real mode into the 80286's protected mode, but going back was a lot more difficult, and this turned out to be a problem in practice), this required a new register for the CPU to keep the instruction pointer when operating in 32-bit mode.
In the 80386, the extended (32-bit) registers were named Exx whereas the corresponding 16-bit registers were as previously referred to as xx. So you'd get for example the old 16-bit accumulator register AX plus the extended 32-bit accumulator register EAX (with 16-bit halves EAH and EAL). In keeping with this naming convention, the Extended Instruction Pointer register was referred to as EIP.
As an aside, most often, the correct value isn't merely EIP, but CS:EIP, because the value of EIP remains an offset from some starting location which is defined by the code selector.