How did Evernote detect the security breach?

Question

http://blog.evernote.com/blog/2013/03/02/security-notice-service-wide-password-reset/

Evernote's official message says little about technical details of the attack and how it was detected.

Questions:

0) Speculate: How did Evernote detect the security breach?

1) How can unauthorized database access with valid credentials be prevented and detected?

2) How can database or file storage data leak be prevented and detected?

If this are too abstract questions, assume:

• This is a rich client web application with database backend and Node.JS in between.
• Apache Cassandra is the primary data store and ElasticSearch is a search index/engine
• SQL Injection attack vector is not applicable or has been mitigated.

Answer 1

1. Speculation: they look to hire people who have experience with:

   • OSSEC (host-based IDS)
   • Tripwire and Samhain (file integrity)
   • Splunk (log aggregation, reporting & alerting)
   • Nagios
   • firewall & threat management
   • grsecurity/pax/suhosin etc
   • Nmap, BackTrack, and Metasploit (proactive security tools)

   When the perimeter was breached, their monitoring and system management worked.

2. Assuming a least-privilege model, non administrative accounts accessing tables/data to which they do not have access (logged violation); or queries which do not match "known good" patterns (adaptive or anomaly detection); or queries which match "known bad" patterns (SELECT * FROM USERS), i.e. signature based IDS; or atypical system load patterns; or atypical data volume per query.

3. DLP, i.e. outbound content filtering/inspection for canary values, or suspicious values (the classic case being strings that are valid credit card numbers or SS numbers)

Answer 2

It was most likely detected by some sort of IDS (which is it's job, after all), or though manual log inspection. This behavior might set off alarms:

• If a database dump was made outside of the regular schedule.
• Logins from unusual IP's.
• Logins at unusual times.
• Unusual traffic spikes.
• By logging network traffic - for example, one could inspect packets for database information that should never leave the servers.
• Accounts performing actions they don't usually perform, or have no right to perform.
• Any unusual behavior really.

After actions are flagged by an IDS, a manual inspection is performed, with varying urgency, depending on how the IDS has classified it.

You can attempt to prevent unauthorized access by implementing a scheme like the above.