Contextual Smart Computer Vision for Live Device Interaction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies lack the capability for live, contextual interactions between multiple digital devices, such as sharing content summaries from a laptop to a smartphone, which is essential for seamless multi-device user experiences.
Innovation Solution
A computerized method using a digital camera on a mobile device to capture images of a laptop screen, employing machine learning algorithms to classify the content and suggest contextual actions based on trained digital images and associated actions, enabling live context sharing between devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If traditional device interaction methods are used, then device functionality is limited to basic operations, but contextual awareness and automation capability are lost
Solution Approach 1:
The system enables devices to automatically capture, classify, and share contextual information without manual intervention. The mobile device's camera automatically captures screen content, the ML classifier automatically categorizes it, and relevant actions are automatically suggested and executed, allowing the system to serve itself rather than requiring continuous user input
Solution Approach 2:
The system implements a feedback loop where the mobile device captures content from the laptop, the ML classifier analyzes it, and the results are fed back to suggest contextual actions. This continuous feedback mechanism enables the system to learn from captured content and improve its contextual understanding over time
2Ease of operation
If manual content sharing between devices is implemented, then basic content transfer is achieved, but live contextual interaction and intelligent suggestions are lost
Solution Approach 1:
The system performs preliminary actions by capturing and classifying content in advance before the user needs it. The mobile device continuously captures screen content and the ML classifier pre-processes this information, so when the user needs contextual sharing, the analysis is already complete and ready for immediate action
Solution Approach 2:
The patent replaces manual mechanical operations (physically transferring files, manually copying content) with an automated optical and computational system. The mobile device's camera optically captures screen content, and ML algorithms automatically process and classify the information, substituting manual mechanical content transfer with intelligent automated processing
3Device complexity
If simple image capture is used, then basic visual recording is achieved, but content classification and action determination are lost
Solution Approach 1:
The system segments the complex task of content understanding into distinct manageable components: image capture by the camera, ML-based classification of the captured content, and determination of contextual actions. This segmentation allows each component to be optimized independently while working together to solve the overall problem
Data Source
AI summary
In one aspect, a computerized method for contextual smart computer vision comprising: with a digital camera of a mobile device, obtaining a digital image of a computer screen, wherein the computer screen is displaying a specified computing application; with a machine learning algorithm: obtaining a set of training digital images of computer screens and associated contextual actions, and using the machine learning algorithm to build and train a machine learning classifier based on the set of training digital images of computer screens and associated contextual actions; and using the machine learning classifier to classify the digital image and determine a specific application action based on the classification of the digital image; and based on context, the mobile application suggests specified contextual actions.