Google has begun the deployment of Gemini’s innovative real-time AI video functionalities, enabling the platform to interpret visual input from a user’s device display or smartphone’s camera and provide contextual answers. Confirmation of this deployment comes nearly a year after the initial demos of “Project Astra,” the underlying technology powering these capabilities, were showcased during Google I/O 2024. Reports from a Reddit userinitially showed the appearance of this feature on a Xiaomi handset. This user subsequently shared a video showcasing Gemini’s new ability to analyze screen content. This screen-reading functionality is one of two key features that Google announced in early March would be progressively made available to Gemini Advanced subscribers with a Google One AI Premium plan later in the same month. The second feature announced uses the smartphone’s camera to process a live video feed, allowing Gemini to understand and respond to questions about the user’s surroundings.
This update follows the recent introduction of Gemini’s Canvas feature, which aids users with writing and coding tasks, and the addition of podcast summarization tools. That update coupled with Gemini Live Video really just shows how far ahead Google is right now when it comes to AI assistant technology when compared to Apple, Samsung, and even Amazon.
Of course, this doesn’t seem to encompass every single aspect of Project Astra as it was demoed last year and can be seen in the above video. The full demo showed off the assistant retaining memory of items it saw via your camera moments ago, and telling you later where it saw it — or even giving you the option to draw on items to ask more specific information on the circled area.
However, to me, it feels like we are not far off from this AI assistant future that we were shown last year. The rate at which Gemini is progressing in its feature set is very promising, and I am very excited to see how Google will top this at this year’s Google I/O conference.
GIPHY App Key not set. Please check settings