If done properly, it could be a somewhat useful addition to your security protocol. However, I'd be very hesitant about using it to replace a pre-existing factor in your authentication, as IP addresses are not secret information (you give it away to every website you visit) and depending on how/where you grab the IP address from, could be trivial to spoof. But if you use the IP address from a TCP connection in conjunction with say a stored secure http-only cookie, you could add some security. Specifically you'd invalidate the cookie whenever the IP address changes (which may change for benign reasons such as your ISP reassigning your IP) as well as malicious reasons (attacker got hold of your cookie).
As Ladadadada said, IP addresses in TCP connections aren't easily spoofed anymore. TCP requires a handshaking procedure to get a random 32-bit sequence number from the server before you can exchange information. If you forge a completely phony random IP address when starting the handshaking procedure, then the packets won't be routed to your computer through the internet, so you can't complete the handshake. Unless, that is you control intermediate routers at the ISP or a computer on one of the same local networks that could capture the packets routed to elsewhere, in which case you could forge random IP addresses.
However, if you design a web app and record the IP address, you have to be careful. Say you have a web app with two web servers (e.g., one for dynamic content/one for static), behind a load balancer, or other proxy. You may see by trial and error that the client's IP address is only present in the HTTP header
X-Forwarded-For. However, this field is easy to change. For example,
telnet www.whatismyip.org 80 and type the following with/without
X-Forwarded-For the line (remember to press enter twice after the last line to indicate the end of your HTTP request).
GET / HTTP/1.1
and you'll see that this web app thinks your IP address has changed to
188.8.131.52. So be sure to test thoroughly. Overall, I think doing this yourself is more work than its worth, especially as it may frustrate users whose IP addresses change quite frequently.
EDIT: I realized after writing this that while I answered the title question ('can IP address be a component of 2 factor auth?"), but not the specific part referring to ssh management. I'd say ssh passphrase protected key is essentially two-factor auth: something you have (the ssh key), and something you know (the passphrase's key).