Are web scrapers fooled by obscured emails anymore? [duplicate]

Question

It's a common practice online that instead of writing your email as [email protected] people will instead write it as someone AT example.com in an attempt to make it harder for web scrapers to find your email address on a web site.

Is this even that effective anymore? I would imagine anyone scraping the web for emails could just as easily check for a pattern like that and transform it to an email address.

I'm sure that some strategies of obscuring the email address are more effective than others, as not every combination that is legible to a human could be accounted for in a program, but common ones like the one I describe above could be found just as easily.

If trying to obfuscate an email, what kind of strategies could be used to ensure that a human reader can understand but a program would not? Couldn't the scraper just continue to be updated to understand new patterns as its author finds them?

Answer 1

You've got multiple methods really, you should of course consider that such bots harvesting this content are essentially scraping whatever pages they come across and searching for patterns that look like email addresses. As you say, it's a bit of an arms race and there's nothing stopping the people developing such scrapers from implementing these methods (wait, is that why you're asking?)

You're going to want to avoid actually creating a hyperlink out of your email address in most cases, and you certainly want to avoid using mailto: - that's basically announcing to anybody reading the page "hey, I'm an email address".

Let's start of nice and simple, spacing:

m y e m a i l @ m y d o m a i n . c o m

It's obviously an email address to a human, looks like a bunch of random letters with spaces to a scraper. Don't like spacing? Much less common but far more foolproof is to convert your email address into an image. It's still human readable but it's not going to be something that most email scrapers are looking for, let alone able to parse.

How about converting your punctuation (@ and period) into their HTML equivalents (@ and . respectively)?

myemail@mydomain.com

This still looks like an email address when rendered by the browser, but it isn't going to be all that difficult to work around from the point of view of scraping since you'd just look for the . and @ - but why stop there? Why not go all the way and just encode the entire email address? This can be done quite easily with a tool like Rumkin's Mailto Encoder, suddenly your email address looks like this:

myema%69l@my%64%6fma%69n%2e%63om

This still renders like you'd expect in a browser, but is basically gibberish as far as any scraper that doesn't take the encoding into consideration.

You can also do this with CSS if you're so inclined with something like this:

<style>
  my-email::after { content: attr(data-domain); } 
  my-email::before { content: attr(data-user); }
</style>

<my-email data-user="myemail" data-domain="mydomain.com">@</my-email>

Or, as already discussed on Stack Overflow, you could just use JavaScript.

Answer 2

Hiding your email using javascript can only get you so far. There are two types of scraping engines that are used to collect data from a website.

Classic: The classic scraper is simply doing a GET request on the url and then parsing the HTML that is returned from the server.

• Advantage: Has the advantage of quick data collection and higher throughput both from a bandwidth and processor perspective.
• Disadvantage: It doesn't actually load the page in the way a browser does. Since there is no DOM loaded, any javascript based content will not be available to the scraper. This means that any of the methods mentioned by Flyk will work great when combatting these scrapers.

Browser Based: The browser based scrapers are a new breed of scraper and allow the engine to actually load the page into a "web browser". (some of these are headless - phantom.js)

• Advantage: This type of scraper has the ability to effectively render a webpage and scrape the results exactly as they would appear to a user. This means that this type of scraper could read any emails that have been encoded with javascript.

• Disadvantage: These scrapers are also much more complex to create and require a longer loading period and more bandwidth before a page can be scraped. For these reasons, many scrapers still just use the classic style of scraping.

In the end, it would be better to use javascript to encode your email address rather than just typing it in plain text. If you really want the best protection for your email, you should go with the method of creating an image of your email address.

Images can be read using OCR but the complexity is well beyond most scraping engines.

Answer 3

One fairly fool proof idea would be to include the e-mail address in an image vs. text. I would imagine this method could be defeated by a program that can read text in images, but it would be much harder to defeat than plain text.

Answer 4

If trying to obfuscate an email, what kind of strategies could be used to ensure that a human reader can understand but a program would not?

An alternate solution (which does not display the email on the page) is to use a contact form with some captcha mechanism to prevent mass mailing.

You could add to this a an automatic reply from a real email address (one which can be saved as a contact).

Answer 5

To be fair, the most secure method is to use an image of your email address as previously stated.

The main downside to the this is that if users have images disabled, they won't see it. This can be counter-acted, however, by in the HTML img alt="" alt tag, placing your email as HTML encoded characters: e.g. myema%69l@my%64%6fma%69n%2e%63om.

Another downside would be that users cannot click on this... simply wrap the image in a mailto link, but that would totally negate the email address hiding from scrapers.