EXHIBITION
2025- MIT Reality Hack
BUILD WITH
adobe creative suit, aivideo, figma, miro, premiere, Unity, Meta Quest
Members: Arthur Baney, Sowilo Xiong, Jiachen Zheng, Ziying Qi, Sutanuka Jashu
Inspiration
Losing essential items is a universal frustration—whether it’s searching for your keys in the morning, managing a shared household inventory, or helping hotels and rental properties keep track of guest essentials. The lack of efficient and intuitive solutions inspired us to create Mr. Finder. By combining spatial memory reinforcement with cutting-edge XR technology, we aim to transform how people organize their spaces and locate lost items, bringing simplicity and peace of mind to everyday life.
What it does
Mr. Finder is a mixed-reality (MR) application that helps users locate misplaced objects by creating a spatially tagged inventory within a mapped environment. Organize Your Space: Users take pictures of objects, categorize them, and tag their locations in a virtual representation of their environment. Find Items Instantly: Use voice commands or a virtual keyboard to search for items. The app guides users with immersive virtual markers and arrows anchored to real-world spaces. Hotel/Host Use Case: For shared spaces, such as hotels or rental properties, hosts can pre-tag items (e.g., remote controls, toiletries, kitchen supplies) to simplify guest interactions and ensure inventory accountability.
How we built it
Technology Stack: Unity with MRTK (Mixed Reality Toolkit): For creating an immersive mixed-reality environment and integrating advanced interactions. Meta Quest All-in-One XR SDK: Leveraged extensively for passthrough capabilities, voice commands, spatial mapping, and real-time interactions. SLAM (Simultaneous Localization and Mapping): Used to create a precise virtual map of the room and anchor objects to real-world locations. Wit.ai Voice SDK: Enabled intuitive voice-based commands for searching, adding, and managing objects within the app.
Development Features: Room Mapping with Pass Through: Using Meta Quest, users map their surroundings, creating a spatial representation of the room with virtual markers anchored to specific locations. Inventory Management with interactive SDK: Add Objects: Users can input objects into the system via preloaded photos or voice commands, categorizing them by location or purpose. Delete Objects: Items can be easily removed or re-tagged as needed to maintain an accurate inventory. Voice Interaction with Wit.ai: Implemented voice recognition for multi-language commands, allowing users to perform actions such as: “Where is my notebook?” “Add keys to the kitchen table.” The system processes these commands seamlessly through the Meta Quest Voice SDK. Building Blocks for UI/UX: Designed intuitive user workflows with Meta Quest’s Building Blocks for interaction, ensuring smooth transitions between actions like tagging, searching, and navigating.
Challenges we ran into
We encountered several challenges during development. The Oculus platform's limitations with hand gestures made it difficult to implement a seamless hands-free tagging process, and saving pre-recorded images often resulted in inaccuracies during spatial mapping. Integrating voice commands with Unity using ChatGPT posed difficulties in enabling smooth speech-to-text functionality, processing user input, and connecting it dynamically to the system API, especially in noisy environments. Additionally, ensuring precise spatial mapping for anchoring virtual tags required iterative refinements, and balancing an intuitive UI with advanced XR features demanded continuous testing to enhance user experience.
Accomplishments that we're proud of
Seamless Room Mapping: Using Meta Quest’s passthrough and SLAM technology, we created an intuitive and accurate system for mapping physical spaces and anchoring virtual tags to real-world objects, making organization effortless. Voice-Driven Interaction: Successfully integrated Wit.ai voice commands, enabling users to locate items naturally with multi-language support. Commands like “Where is my notebook?” ensure quick and easy item retrieval. Collaborative Inventory Management: Designed a multi-user system where households, teams, or rental properties can share inventory access, enhancing collaboration and functionality. Cognitive3D Integration: Implemented Cognitive3D’s SDK for tracking and labeling saved items within a mapped environment. This not only enhances real-time organization but also provides data analytics stored in a cloud system, paving the way for potential digital twin integration in the future. Scalable Applications: Showcased the versatility of Mr. Finder for personal organization, household use, and professional scenarios, such as inventory tracking in hotels or rental properties.
What we learned
This journey taught us the importance of creating human-centered solutions. We learned that advanced technologies like XR and voice recognition must integrate seamlessly into everyday life to truly serve their purpose. Tackling challenges like platform limitations and spatial accuracy taught us the value of persistence and iterative testing. Above all, we realized that simplicity, clarity, and adaptability are key to building a tool that’s both innovative and practical.
What's next for MR.Finder
The future of Mr. Finder is filled with exciting possibilities:
AI-Driven Object Recognition: Automate tagging by allowing the app to recognize and categorize objects without manual input. Enhanced Collaboration: Expand multi-user capabilities to enable shared inventory management across households, offices, and other spaces. Sustainability Features: Integrate options for decluttering, such as suggesting donation or recycling for underutilized items, while tracking the environmental impact of organizing habits. Digital Twin Integration: Explore using Cognitive3D’s analytics platform to create digital twins of user environments for advanced inventory management. Cross-Platform Support: Extend compatibility to more XR devices, ensuring accessibility for diverse users and scaling the application across industries. Also a future possibility to add the new Convai for interactive realtime digital twin assistance for future development.
