almost a decade Google recently showed off a feature called Now on Tap in Android Marshmallow: tap and hold the home button and Google will display useful contextual information related to what’s on the screen. Talk about a movie with a friend over text? Now on Tap could give you details about the title without having to leave the messaging app. Are you searching for a restaurant on Yelp? The phone could show OpenTable recommendations with just a tap.
I was just out of college and found these improvements exciting and magical – its ability to understand what was on the screen and predict actions you might want to take felt forward-looking. It was one of my favorite Android features. It slowly morphed into Google Assistant, which was cool on its own, but not exactly the same.
Today, at Google’s I/O developer conference in Mountain View, California, the new features Google is touting in its Android operating system feel like the Now on Tap of yesteryear: They let you take advantage of the contextual information around you to Make using your phone a little easier. easier. Except this time, these features are powered by a decade of advances in large language models.
“I think what’s exciting is that we now have the technology to create really interesting assistants,” Dave Burke, vice president of engineering at Android, tells me during a Google Meet video call. “We need to be able to have a computer system that understands what it sees and I don’t think back then we had the technology to do it well. Now we do it.”
I had the opportunity to speak with Burke and Sameer Samat, president of Google’s Android ecosystem, about what’s new in the world of Android, the company’s new Gemini artificial intelligence assistant, and what all this holds for the future of the operating system. Samat called these updates a “once-in-a-generation opportunity to reimagine what the phone can do and rethink everything about Android.”
Circle to search… your task
It starts with Circle to Search, which is Google’s new way of approaching Mobile Search. Like the Now on Tap experience, Circle to Search, which the company introduced a few months ago, is more interactive than simply typing in a search box. (You literally circle what you want to look for on the screen.) Burke says, “It’s a very visceral, fun, modern way to search… It’s also younger because it’s so fun to use.”
Samat says Google has received positive feedback from consumers, but the latest Circle to Search feature comes specifically from student comment. Circle to Search can now be used on physics and math problems when a user circles them; Google will show step-by-step instructions to complete the problems without the user leaving the syllabus app.
Samat made it clear that Gemini was not only providing answers but also showing students how to solve problems. Later this year, Circle to Search will be able to solve more complex problems, such as diagrams and graphs. All of this is powered by Google’s LearnLM models, which are optimized for education.
Gemini becomes more contextual on Android
Gemini is Google’s AI assistant that, in many ways, outshines Google Assistant. Actually, when you activate Google Assistant on most Android phones these days, there is an option to replace it with Gemini. So naturally, I asked Burke and Samat if this meant that the Wizard was heading to the Google Graveyard.
“The way to look at it is that Gemini is an optional experience on the phone,” Samat says. “I think obviously over time Gemini is becoming more advanced and evolving. We don’t have anything to announce today, but consumers have a choice if they want to opt for this new AI-powered assistant. You can try it and we are seeing that people are doing it and we are receiving very good comments.”