When Google Lens was introduced in 2017, the search function accomplished a feat that not long ago would have seemed like the stuff of science fiction: Point your phone’s camera at an object and Google Lens can identify it, show some context, maybe even leave it alone. . you buy it. It was a new way of searching, one that didn’t involve clumsily writing down descriptions of the things you saw in front of you.
Lens also demonstrated how Google planned to use its machine learning and artificial intelligence tools to ensure its search engine appears on every possible surface. As Google increasingly uses its core generative AI models to generate information summaries in response to text searches, Google Lens visual search has also been evolving. And now the company says Lens, which powers about 20 billion searches per month, will support even more forms of search, including video and multimodal searches.
Another tweak to Lens means even more context for purchases will appear in the results. Unsurprisingly, shopping is one of Lens’ key use cases; Amazon and Pinterest also have visual search tools designed to drive more purchases. Search for your friend’s sneakers on the old Google Lens and you may be shown a carousel of similar items. In the updated version of Lens, Google says it will show more direct links to purchases, customer reviews, editor reviews, and comparison shopping tools.
Lens search is now multimodal, a buzzword in AI these days, meaning people can now search with a combination of video, images, and voice input. Instead of pointing their smartphone camera at an object, tapping the focus point on the screen, and waiting for the Lens app to get results, users can point the lens and use voice commands at the same time, e.g. “What kind of “What are those clouds?” or “What brand of sneakers are those and where can I buy them?”
Lens will also begin work on real-time video capture, taking the tool a step beyond identifying objects in still images. If you have a broken record player or see a flashing light on a malfunctioning appliance at home, you can record a quick video through Lens and, through a generative AI overview, see tips on how to repair the item.
First announced at I/O, this feature is considered experimental and is available only to people who have opted in to participate in Google Search Labs, says Rajan Patel, Googler of 18 years and co-founder of Lens. The other Google Lens features, voice mode and expanded shopping, are rolling out more widely.
The “video understanding” feature, as Google calls it, is intriguing for several reasons. While it currently works with videos captured in real time, if Google expands it to captured videos, entire repositories of videos, whether on a person’s own camera roll or in a gigantic database like Google, could potentially be tagged and be overwhelmingly affordable.
The second consideration is that this Lens feature shares some features with Google’s Project Astra, which is expected to be available later this year. Astra, like Lens, uses multimodal inputs to interpret the world around you through your phone. As part of an Astra demonstration this spring, the company showed off a pair of smart glasses prototypes.
On the other hand, Meta just made a splash with its long-term vision for our augmented reality future, which involves mere mortals wearing silly glasses that can intelligently interpret the world around them and show them holographic interfaces. Google, of course, has already attempted to make this future a reality with Google Glass (which uses fundamentally different technology than the last Meta release). Are the new Lens features, along with Astra, a natural transition to a new type of smart glasses?
