If an attacker turns on wifi but doesn't have the security key to connect to an access point in range, can he still sniff packets that travel between the access point and clients connected to the access point, and thus get the Mac addresses of the clients? If the access point is a public wifi and there is no security key but there is Mac filtering, does that make a difference?
An attacker can always determine the client's MAC address if they can sniff packets to or from the client. This is true regardless of whether encryption is used or not. The MAC address is in the outer encapsulation layer of the 802.11 packet, and there is no encryption applied to that level. Here's a good link at Microsoft that lays out the packet encapsulation, including where encryption happens in 802.11.
This is kind of the expected result. By definition, the physical and data link layer information has to be openly available to other network devices so that they all can figure out who's supposed to send what where.
Standard tools like Netstumbler will display MAC addresses for you. Your followup question will be "But doesn't that make it trivial to bypass MAC address filtering as a security measure on the AP?" And the answer is, yes. Yes it does.
As @Gowenfawr has the answer nailed I'll just focus on that final bit of your question.
Mac filtering doesn't really make a difference if someone actually wants to connect.
All it is good for is stopping devices connecting automatically, which can sometimes help with load and contention, but really - do not rely on it for anything...you need stronger authentication, such as a security key or an end to end VPN.
Have a look at this CNN article about the Wifi network at Decon for some scary reading:-)
From the client definitely yes, an attacker can sniff packets. This is not dependent or related to the MAC address at all. The OSI model bi-furcates the physical access and makes it openly accessible, which means that this is possible. Said so, there are ways as well to bypass filtering for MAC addresses on different access points making it a little difficult to identify and work with from the attacker's perspective.
There is so many solution for that. But yes. A hacker would use packet sniffing to find out Mac address. Even if you use encapsulation or encryption. Just the wasted time is different. The best way is to keep WiFi range inside your required room.