Regarding privacy, the purpose of fixing a number to your house is to inform people of the number. Granted, not necessarily to inform Google in particular. I don't think there's anything inherently wrong with working out the number of a house from a picture of that house.
I think the ethical issue of Google using humans to identify these numbers, or for that matter OCR-ing them, is subsumed by the ethical issue of whether it was OK in the first place for Google to compile all these images that can be obtained by passers-by in public streets.
Now, I'm not going to rule on whether it's ethically OK for Google to aggregate millions of photos of people's houses. Some folks aren't happy about it. If Google agrees for whatever reason to remove your house from StreetView, then I'd hope that they also don't use it in StreetView-based CAPTCHAs.
There's maybe an entirely separate ethical issue, what effort it's OK to make CAPTCHA-completers do on your behalf, and to what productive purpose. One could imagine a Mechanical-Turk based CAPTCHA system that just assigns any old kind of HIT to people -- one that has been done before to verify they're human, and one that hasn't for your personal profit. This might well be regarded as unethical, or at least chiselling. So I suppose one could argue on that basis that it's not OK for Google to use CAPTCHAs in order to locate houses on StreetView for the benefit of their mapping products. reCAPTCHA also helps scan books, so the same ethical issue applies there. I think most people just figure that if you want to log into somebody's website, or use their CAPTCHA product, then they can ask you to do some small amount of work for them in return.