Stealing Passwords from Finger Movements Using Meta Ray-Ban
機械翻訳 / Machine-translated
A
AI Friends
1日前・15分で読める
機械翻訳 / Machine-translated
@aifriends
AI Friends（https://aifriends.jp）のクロスポスト公式アカウント。AIツールの紹介・使い方・できることを、中学生でもわかるやさしい日本語で届けます。
What if the glasses worn by the person sitting across from you at a café actually had a camera hidden in them? Researchers at Tel Aviv University have published a method for reading finger movements from smart-glasses footage to crack smartphone passwords. This article explains how it works and what you can do to protect yourself.
This research was reported in a paper titled "Watch and Crack," published on arXiv. The paper was submitted on September 6, 2026.
The authors are Yoav Orenbach and Avishai Wool. According to reporting by ITmedia, the institution is Tel Aviv University in Israel.
The device used is Meta Ray-Ban smart glasses. They look like ordinary eyewear but contain a camera.
An attacker wearing these glasses films the hands of someone operating a smartphone, then analyzes the footage to infer what the person typed on screen.
This kind of eavesdropping has long been called "shoulder surfing." What makes this approach distinctive is that the attacker does not need to stand behind the target — they can aim from directly across the table.
According to ITmedia's explanatory article, the attack combines multiple AI systems.
First, an image-recognition AI (SAM 3) and a hand-tracking model (MediaPipe Hands) follow the fingertips at sub-pixel precision — finer than a single pixel.
Next, a 3D neural network (an AI that processes video as a temporal sequence) identifies the exact moment each key is pressed.
Backspaced corrections — where the user types, makes a mistake, and deletes it — are automatically detected and filtered out.
Even when the screen is not visible, the attack still works. By observing where the shift key and layout-switch keys are pressed, the system reverse-engineers the boundaries of the keyboard.
Prior knowledge of the target's phone model or keyboard app (Gboard, iOS, Samsung) is reportedly not required. The method is said to handle passwords that mix uppercase, lowercase, digits, and symbols.
What the footage reveals is a probability — "this key was most likely pressed" — not a certainty.
To narrow things down, the paper's abstract states that statistics from 2.19 billion leaked passwords are combined with the visual data. By referencing common character sequences that people tend to choose, the candidate list is whittled down.
For example, if the footage only narrows a character down to "5 or 6," frequency patterns in real-world passwords can suggest which one is the stronger bet.
According to ITmedia, in an experiment with 16 participants, keystrokes were correctly identified in 77.8% of cases overall.
The attack was effective at filming distances of up to 1.8 m. Results were confirmed on three device models: Pixel 10, iPhone 16, and Galaxy A53.
The paper's abstract provides even more specific figures.
Human-chosen passwords that conform to corporate password policies — up to 16 characters long — could reportedly be cracked within hours to days.
Even randomly generated passwords created by a password manager could be broken in under an hour on modern GPUs, provided they were 13 characters or fewer.
Compared with baseline methods, the attack was able to reduce password entropy (a measure of how hard a password is to predict) by up to 60 bits.
According to the ITmedia article, success is not guaranteed in every situation.
Accuracy drops for younger users who type with very small, fast finger movements. The same applies to people who type hesitantly or those who tap with both thumbs nearly simultaneously.
This is laboratory research. It does not mean that everyone in public is at immediate risk.
Even so, the demonstration that typed content can be inferred from finger movements alone carries significant implications.
The most common defense against smartphone shoulder surfing is a privacy screen protector, which makes the display unreadable from the side or at an angle.
But this attack targets the fingertips, not the screen. The ITmedia article notes that screen protectors and privacy display features may be bypassed if the finger movements are captured on video.
The contrast can be summarized as follows:
Glasses-style cameras are also much harder for bystanders to notice. A Gizmodo Japan article points out that because they look like ordinary glasses, people rarely realize filming is taking place. A recording indicator light exists, but few people are likely to notice it.
In Japan, smartphones are used in public constantly — on commuter trains, in cafés, in the facing seats of bullet trains, and more.
Imagine a company employee at a café during a business trip, opening their laptop and logging into a work service on their smartphone. Simply seeing someone across the table wearing glasses would not raise any red flags.
This paper does not include specific countermeasures. The following are general self-defense measures and are not part of the paper's content.
With multi-factor authentication in place, even a stolen password does not immediately lead to unauthorized access. Start there.
This is laboratory research. That said, the fact that it was demonstrated using commercially available smart glasses is not something to dismiss lightly.
Not entirely useless. However, the ITmedia article notes that they may be bypassed if finger movements are captured on video. Relying on a screen protector alone is risky.
The effect was confirmed on the Pixel 10, iPhone 16, and Galaxy A53. It is not a problem limited to any particular model.
Yes. The system reverse-engineers the keyboard's boundaries by observing where keys like the shift key are pressed.
Reduce how often you type passwords by hand. Using a password manager or passkeys minimizes the situations in which you are a viable target.
Start by enabling multi-factor authentication on your most important accounts.
This article is a cross-post from AI Friends.
