VueBuds are Camera-Equipped Sony WF-1000XM3 Wireless Earbuds That Can See the World Around You

Maruchi Kim led a team at the University of Washington in a project that quietly rewrites what wireless earbuds can do, called VueBuds. They started with a familiar pair of Sony WF-1000XM3 earbuds and turned them into devices that capture images from the wearer’s perspective while staying true to their original size and comfort.

Kim, a doctoral student, collaborated with Professor Shyam Gollakota and five other researchers to solve a puzzle. Every day, we wear earbuds next to our eyes, yet they only play sounds. So the team wondered if they could incorporate a tiny camera inside one of these things without causing the battery to deplete faster or impacting how well it fit in our ears. They designed a little module approximately the size of a grain of rice and placed one in each earphone. Custom 3D-printed shells hold everything neatly in place while allowing you to utilize the original charging case normally. The cameras draw power directly from the earphones and turn off until you need to use them, limiting battery consumption to little under 5 milliwatts when in use.

They can record in black and white at a somewhat low resolution, but the results are still quite impressive. Each camera is oriented slightly outward, by five to ten degrees. When both earphones begin sending photographs back via Bluetooth to a phone or other device, the software stitches them together to form a single view. That combined view covers around one hundred degrees straight in front of you, allowing you to read signs, notice things, and follow a path with ease, but portion of what each lens sees is covered by your face. Even though the cameras are inside the earbuds, they look and feel exactly like ordinary earbuds.

Once the merged image reaches your device, a local vision language model takes over. Nothing is transferred to the cloud or stored anywhere, so you can simply ask it a question and it will respond within a second or so. Do you want to know what the label on a food can says? The technology can read calories or ingredients aloud for you. When you see something weird on a workbench, it will inform you what it is and what it is used for. Travelers can point their earbuds at a street sign in a language they don’t understand and receive an instant translation. They’re also great for fast reminders, like locating a certain item on a cluttered store rack.

People that got to try the technology out particularly liked how natural the interaction seemed. They had 90 individuals complete 17 different vision-related tasks, and the results were comparable to those from the Ray-Ban Meta smart glasses. The earbuds handled ordinary tasks including text reading, object recognition, and even basic reasoning with equal success. People don’t have to be as concerned about their privacy as they would be with always-on wearable cameras because the cameras only turn on when you tell them to and don’t keep anything. Your hands remain free even when you’re up to your elbows in plumbing, wiring, or whatever.

The hardware isn’t too expensive, with each camera sensor costing less than a $1 in bulk, and the company believes the entire change will add less than a few dollars to the price of a high-end earphones. The research demonstrates that visual intelligence does not require a new type of device or a pair of bulky frames, but rather can be integrated into something you most likely already own and use. The following ones may improve the frame rate, make the vision larger, or allow you to utilize more spoken commands.
[Source]