Smart Device Ambient Light Sensors Pose Risk to Image Privacy

In George Orwell’s novel “1984,” Big Brother watches citizens through two-way television-like telescreens to monitor them without cameras. Similarly, our current smart devices contain ambient light sensors, which opens the door to a different threat: hackers.

These passive and seemingly harmless smartphone components receive light from the environment and adjust the screen brightness accordingly, like when your phone automatically dims in a bright room. However, unlike cameras, apps don’t need to ask permission to use these sensors.

In a surprising discovery, researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) found that ambient light sensors are vulnerable to privacy threats when integrated into the display of a smart device.

The team proposed a computational imaging algorithm to recover an image of the environment from the screen’s perspective using subtle changes in the light intensity of a single point from these sensors to demonstrate how hackers could use them in conjunction with monitors.

The paper was published in Scientific advances at the beginning of this January.

“This work turns the ambient light sensor and screen of your device into a camera! Ambient light sensors are small devices implemented in almost all wearable devices and screens that surround us in our daily lives,” says Felix Heide, professor at Princeton University, who was not involved in the study. paper. “As such, the authors highlight a privacy threat affecting a broad class of devices that has until now been overlooked.”

While phone cameras have previously been exposed as security threats for recording user activity, the MIT group found that ambient light sensors can capture images of users’ touch interactions without a camera. According to their new study, these sensors can listen to common gestures, such as scrolling, swiping, and swiping, and capture how users interact with their phones while watching videos. For example, apps with native access to your screen, including video players and web browsers, could spy on you to collect this data without permission.

According to the researchers, the common belief is that ambient light sensors do not reveal significant private information to hackers, so there is no need to program applications to request access to them. “Many believe that these sensors should always be on,” says lead author Yang Liu, MIT Department of Electrical Engineering and Computer Science (EECS) and Ph.D. from CSAIL. student.

“But like the telescreen, ambient light sensors can passively capture what we’re doing without our permission, while apps must request access to our cameras. Our demonstrations show that when combined with a display, these sensors could represent “some sort of threat to the privacy of the images by providing that information to hackers monitoring your smart devices.”

Collecting these images requires a dedicated inversion process in which the ambient light sensor first collects low-bitrate variations in light intensity, partially blocked by hand contact with the screen. The outputs are then mapped to a two-dimensional space forming an inverse problem with knowledge of the screen content. An algorithm then reconstructs the image from the perspective of the screen, which is iteratively optimized and removed using deep learning to reveal a pixelated image of hand activity.

The study presents a novel combination of passive sensors and active monitors to reveal a previously unexplored imaging threat that could expose the environment in front of the screen to hackers processing sensor data from another device. “This threat to image privacy has never been demonstrated before,” says Liu, who worked alongside Frédo Durand on the paper, MIT EECS professor, CSAIL member and lead author.

The team suggested two software mitigations for OS vendors: tighten permissions and reduce sensor accuracy and speed. First, they recommend restricting access to the ambient light sensor by allowing users to approve or reject those app requests. To further prevent any privacy threats, the team also proposed limiting the sensors’ capabilities.

By reducing the accuracy and speed of these components, the sensors would reveal less private information. From a hardware point of view, they argued that the ambient light sensor should not be directly in front of the user on any smart device, but rather placed on the side where it will not capture any meaningful touch interaction.

Getting the image

The investment process was applied to three demonstrations using an Android tablet. In the first test, the researchers sat a mannequin in front of the device, while different hands made contact with the screen. A human hand pointed at the screen, and later, a cardboard cutout that looked like an open-hand gesture touched the monitor, and pixelated fingerprints collected by the MIT team revealed physical interactions with the screen.

A later demonstration using human hands revealed that hackers could gradually capture the way users swiped, scrolled, pinched, swiped and rotated using the same imaging method, although only at a rate of one frame every 3.3 minutes. . With a faster ambient light sensor, malicious actors could spy on users’ interactions with their devices in real time.

In a third demonstration, the group found that users are also at risk when watching videos such as movies and short clips. A human hand moved in front of the sensor as scenes from Tom and Jerry played on the screen, with a whiteboard behind the user reflecting light onto the device. The ambient light sensor captured subtle intensity changes for each video frame and the resulting images exposed touch gestures.

While vulnerabilities in ambient light sensors pose a threat, this type of hacking is still restricted. The speed of this privacy issue is low, as the current image retrieval rate is 3.3 minutes per frame, which overwhelms the duration of user interactions. Additionally, these images are still a bit blurry if obtained from natural video, which could lead to future research. While telescreens can capture off-screen objects, this image privacy issue is only confirmed for objects that make contact with a mobile device’s screen, much like how selfie cameras cannot capture off-screen objects. of the frame.