72

For instance, would it be possible for an app to determine what pixel range on a smartphone display a user is looking at by analysing their eyes with the front facing camera? If so, with what kind of precision?

It would be very discomforting to know that apps could collect data in the background on how your eyes respond to displaying certain advertisements.

  • 3
    A spyware don't need to go through those trouble. It can silently make a screen shot of what you are looking at. – mootmoot Nov 16 '17 at 11:35
  • 41
    I dont think that's what the OP means @mootmoot . They are wanting to know could it know which parts of the screen you were looking at - so they could for example know which adverts attracted your eye – ISMSDEV Nov 16 '17 at 11:41
  • 2
    Indeed, I'm not talking about copying the user's screen but rather analysing the specific content they are looking at in real time which is a whole other thing. – Laurent Nov 16 '17 at 11:45
  • 2
    I wonder whether this kind of technology would be accurate enough to see your password when using an on screen keyboard – user1 Nov 16 '17 at 14:36
  • 2
    @user1 Using secondary sensors as a means of determining user input with fair accuracy is definitely not an abstract risk. The accelerometers in smartphones can theoretically be used to reconstruct typing on phone keyboards and even keyboards the phone is adjacent to on a desk. See wired.com/2011/10/iphone-keylogger-spying cs.swarthmore.edu/~aviv/papers/aviv-acsac12-accel.pdf and techradar.com/news/… – pwdst Nov 16 '17 at 18:25
85

Eye Tracking for Everyone. 2176-2184. 10.1109/CVPR.2016.239. (2016) - Krafka, Khosla, Kellnhofer et al

Our model achieves a prediction error of 1.71cm and 2.53cm without calibration on mobile phones and tablets respectively. With calibration, this is reduced to 1.34cm and 2.12cm.

So yes - it is possible. This particular study was performed using iOS and achieved a read rate at 10–15fps.

There are several companies selling products with similar technology - UMoove for example. It would not surprise me if a higher precision than in the mentioned paper could be achieved.

If you are paticularly concerned a number of smartphone camera covers are available - here is one example.

  • Good answer! It would be nice though, if you could add some more context from the paper and the names and title of the paper itself in case the link goes dead. – Tom K. Nov 16 '17 at 11:41
  • 34
    @Laurent - Personally if ad providers used this technology i'd be far more concerned about them taking pictures of your face if they have access to the camera. Facial recognition would be a possible way for them to unify your data between devices when you have taken explicit steps to prevent this otherwise. – Hector Nov 16 '17 at 12:07
  • 1
    @Hector That's a valid argument. However, don't you think there are easier ways to identify a single person using multiple devices without using any camera? I reckon somebody who really wants to stay anonymous will allow camera access to apps. But seeing as most people don't know about any of this (or simply don't care), advertisement networks will have plenty of pictures on them already. The only way for them to improve is by gathering new kinds of data. – Laurent Nov 16 '17 at 12:30
  • 18
    @Laurent - online political activist uses a different device to avoid tying to real life identity. Add nothing from their personal life to it, tunnel all connections through a VPN. They have facebook & twitter apps installed to post to social media - both request access to the camera as they allow sharing of photos. Maybe our user has shared a photo from a political rally. If these companies used facial recognition during normal use then unknown to the user facebook/twitter could have paired their alias with their real name. And this could be requested by state authorities. – Hector Nov 16 '17 at 12:56
  • 2
    Note I: For this to happen the app needs to be able to access the camera. Note II: I assume a higher permission can be achieved on iOS than on Android because Android devices come in way more different form factors and thus makes it more difficult to optimize the algoritmes to work on all those devices. – Rolf ツ Nov 16 '17 at 14:28
3

Some 20 years ago Canon implemented in their yet analogue SLR cameras the autofocus system, that focuses on the object the photographer was looking at. The user had feedback what direction was chosen.

So yes, it is possible to determine it, even if the accuracy may be questionable. There can be adviced to deny camera permission to all applications​ but those that really need it.

  • An eye on a viewfinder is a different thing than a pair of eyes glancing at a screen from distance, though. – Bergi Nov 16 '17 at 18:58
  • 1
    Yes, they are different. But it is not a contradiction. – Poutnik Nov 16 '17 at 19:55
  • The autofocus does not track the photographer's eyes. In the past it would simply use the autofocus point selected by the photographer, usually the one in the middle of the viewfinder. More recently the cameras are able to choose whichever autofocus point they think gives the best result, using a computer-based algorithm. I suggest posting on photography.stackexchange.com for more information. – Micheal Johnson Nov 17 '17 at 8:40
  • 1
    What you have written is generally true, but not in this case. This system explicitly chose the point selected by the eye. – Poutnik Nov 17 '17 at 8:44
  • 1
    Just a link to verify that this is actually true dpreview.com/articles/6531126959/… – qwazix Nov 17 '17 at 18:35
2

As Hector answered it is certainly possible, however iOS and latest Android allow you to disable permissions for an app. This (in the absence of system vulnerabilities) would prevent any such app using the camera.

As a general rule giving an app's requested permissions a once over is good practice. If your free card game needs access to your camera then it's probably not the right one to install.

  • While true, some of the apps that would be able to most directly use this eye-ad info would be things like facebook, etc, which most people would naturally grant camera access to, since it does also include camera features. – Geobits Nov 16 '17 at 20:23
  • Facebook doesn't need eye tracking on mobile...they can just record the amount of time each post spends on screen. – ste-fu Nov 16 '17 at 20:43
  • Being on screen is not the same as being looked at. Facebook and its advertisers would be very interested in knowing which ads people actually look at. – Stig Hemmer Nov 17 '17 at 8:22
  • 2
    @StigHemmer Correct on a monitor, but on even a decent sized mobile there is normally only one post that is properly visible. Plus they can - and do - (it's data they give to their paying customers) record the scrolling speed and any variations to get a very accurate picture of how long you spend looking at any one post – ste-fu Nov 17 '17 at 9:49
  • @ste-fu surely it shows both posts and ads at the same time? If so, how does it know which one you're looking at? – Kat Nov 18 '17 at 18:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.