I found an unknown MAC address connected to my home router. How would I figure out what OS the IP or MAC address is running?
As @logicalscope says, you can also look at the MAC lookup chart to find out the manufacturer. This could be spoofed.
You could try to find out the IP address of the device, then you could run nmap against it to see if you can identify the OS.
This could be hidden from you, so an alternative would be to try and connect to it - does it give banners back? Ports 22, 23, 80, 443 etc could be useful starting points here.
Failing that, try forcing it off the network and see if anything breaks :-)
One way to possibly figure out what device it is, is to examine the MAC address itself. The first 3 hexadecimal numbers can be used to determine which company the MAC was assigned to. For instance, 00-03-93 is one (of many) MAC OUI prefixes that belongs to Apple.
This doesn't always work if the MAC has been spoofed, has been set manually, or if it is assigned to a virtual machine (or you have a lot of devices. :-)
You can check the first few digits of the MAC address at http://standards.ieee.org/develop/regauth/oui/oui.txt That'll give you the vendor of the device's NIC.
If you have access to the ARP table on your router you can translate the MAC address to an IP address. Often with 'arp -a' or 'show arp' an a router commandline
On your PC you can check hostname of the corresponding IP and whois information:
Check if you can find out the current connections to your router with a netstat-like command. Check http://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xml what the port may be used for.
This should give you a hand full of leads to further analyse