Real-Time Object Mapping for Arbitrary Input Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
User interaction with computing devices is limited to available input devices within reach, restricting interaction experiences, especially on mobile devices, and does not allow the use of arbitrary objects as input devices for tangible interaction.
Innovation Solution
A system and method for real-time object mapping using cameras to recognize objects, determine their properties, and generate spatial and interaction data, allowing any object to be used as a preferred input device, such as a mouse or keyboard, by tracking user actions and providing these data to a computer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional input devices (mouse, touchpad) are used, then basic interaction functionality is provided, but interaction richness and user experience are limited
Solution Approach 1:
The system enables any physical object to function as an input device by mapping it to virtual input device functionalities. The camera captures images, object recognition identifies the object, and the system determines properties that enable the object to provide spatial data and interaction data, allowing a single system to perform multiple input device functions through different objects.
Solution Approach 2:
The system creates a virtual copy of input device functionality by mapping physical objects to virtual input devices. Instead of requiring physical copies of mice or keyboards, the system recognizes objects and generates corresponding spatial and interaction data that replicates the behavior of traditional input devices.
2Adaptability or versatility
If additional specialized input devices are carried to enrich interaction, then interaction experience improves, but portability and convenience deteriorate
Solution Approach 1:
The system allows everyday objects to serve as input devices, eliminating the need to carry specialized equipment. Any object within the camera's field of view can be mapped to provide input functionality, making the system universally applicable without adding physical weight.
3Speed
If object mapping is performed in real time, then interaction responsiveness is improved, but processing complexity and computational requirements increase
Solution Approach 1:
The system performs object recognition and property determination in real-time as images are captured. By continuously monitoring the environment and pre-processing object data, the system maintains responsiveness while managing computational complexity through efficient real-time processing pipelines.
Solution Approach 2:
The system replaces physical input device mechanics with computational processing. Instead of relying on physical sensors in traditional input devices, the system uses computer vision and object recognition algorithms to detect and interpret object properties, converting physical object characteristics into digital input signals.
4Ease of operation
If arbitrary objects are used as input devices, then ease of operation and convenience are improved, but measurement precision and object identification accuracy may worsen
Solution Approach 1:
The system presents identified objects to the user for selection and validation. This feedback loop allows the user to confirm or correct object identification, ensuring accuracy while maintaining the convenience of using arbitrary objects. The system adapts based on user input to improve identification precision.
Data Source
AI summary
Systems and methods are described for mapping input devices in real time. The method may include controlling a camera to capture a plurality of images, performing object recognition on the images to identify a plurality of objects, determining properties for each identified object that enable the corresponding object to provide spatial data and interaction data of a preferred input device, ranking the objects according to the determined properties, presenting the identified objects in an order corresponding to the rank that enables a user to select one of the presented objects, tracking the selected object and a human being, generating spatial data and interaction data from a result of the tracking, and providing the generated spatial data and interaction data to a computer to emulate the events of the preferred input device.