Native App UI Interaction Capture via Frame Bundle Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for capturing and evaluating user interactions with native application interfaces are inefficient, often requiring significant bandwidth and memory resources, and fail to provide effective tools for modifying interfaces to improve user experience.
Innovation Solution
The method involves generating and transmitting frame bundles that include data on the presentation position and drawing operations of user interface objects, allowing for the recreation of visual changes and user interactions, with instrumentation code using an offscreen PDF drawing context to reduce data transmission and storage needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If screenshots or video are transmitted to capture user interactions, then visual data is captured, but bandwidth and memory resources are significantly consumed
Solution Approach 1:
The patent extracts only the essential visual information (drawing operations, presentation positions, and visual attribute changes) from the complete screenshot or video data. Instead of transmitting entire frame images, it transmits only the differential drawing operations that caused visual changes, significantly reducing data volume while preserving the ability to reconstruct and analyze user interactions.
Solution Approach 2:
Instead of capturing complete visual frames and then analyzing changes, the patent inverts the approach by capturing only the drawing operations and visual attribute changes directly from the rendering pipeline. This allows reconstruction of visual changes without ever creating or transmitting full screenshots, fundamentally reducing the data capture methodology.
2Measurement precision
If complete user session data is captured for evaluation, then interaction analysis is thorough, but data processing and storage complexity increases
Solution Approach 1:
The system extracts only the essential elements needed for interaction analysis: drawing operations, presentation positions, and visual attribute changes. By filtering out redundant information (such as unchanged UI elements and unnecessary metadata), the patent reduces data processing complexity while maintaining sufficient detail for thorough interaction evaluation.
Solution Approach 2:
The patent segments user session data into discrete, manageable units (individual drawing operations and visual attribute changes) that can be processed and stored independently. This segmentation allows for efficient indexing, querying, and analysis of specific interaction events without requiring processing of entire session recordings, reducing overall system complexity.
3Manufacturing precision
If detailed drawing operation data is stored, then interface recreation is accurate, but storage requirements increase
Solution Approach 1:
The patent creates a simplified copy of the visual interface state by recording drawing operations and visual attribute changes rather than storing complete interface snapshots. This copy contains sufficient information to accurately recreate the interface visual changes while occupying minimal storage space, as it only captures the essential transformation data needed for reconstruction.
Solution Approach 2:
The patent stores changes in visual parameters (position, size, color, opacity) rather than complete parameter sets for each UI element. By recording only the delta changes in these parameters between states, the system maintains high interface recreation accuracy while significantly reducing the storage requirements compared to storing complete interface state descriptions.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for evaluating interactions with a user interface of an application are disclosed. In one aspect, a method includes receiving frame bundles for a user session with a native application. Each frame bundle can include data specifying, for each of one or more points in time, a presentation position of each presentation object used by the native application to generate a user interface of the native application at the point in time, and, for one or more presentation objects, one or more drawing operations performed to generate the visual representation of the presentation object. Playback data that presents visual changes of the user interface corresponding to the drawing operations performed to generate the visual representation of each presentation object is generated based on the data specified by the frame bundles.