Orb Agent

Code
Orbs listening while the orb is held.Orbs answering "Can I make it to dinner in Bern?" with a spoken line, a calendar card and a train card.Orbs idle, inviting you to hold the orb and ask about what is on screen.

An iPhone voice assistant for older people. Show a screen, say one sentence, and get a spoken answer with a few cards.

Result
2nd place
Event
Tech Europe Agentic AI Hack Zurich, sponsored by Google DeepMind
Team
Four, built in one day
Built with
SwiftUI, FastAPI, Gemini Interactions API, Gradium voice
My role in the team
Missing, to be added

Phones assume you can read a dense screen, find the right app and tap through it. That is a poor fit at arm’s length, and a poor fit when the thing you care about is already on screen: a message, a calendar, a ticket.

Orb Agent takes that screen and one spoken sentence and answers out loud. A screenshot arrives from the Action Button or with the question. A FastAPI backend sends the words and the screen to a Gemini agent, which replies with one spoken line and up to three cards: what is on the calendar, whether a plan fits, a train, a flight or a link.

Calendar, Gmail, Tasks and Drive are read-only. A change is only ever a suggestion that waits for a confirmation. Someone in the family sets up the server and the token, so the person using the phone never handles keys.

We built it as a team of four in one day and took 2nd place at the Tech Europe Agentic AI Hack in Zurich, sponsored by Google DeepMind.