Mobile AR Occlusion via Feature Point Meshing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional object occlusion solutions for mobile rendered augmented reality environments are incompatible across various mobile computing clients and operating systems, and often require significant processing power and memory, limiting accessibility to users without LiDAR sensors.
Innovation Solution
A configuration that enables object occlusion in mobile AR game systems across multiple generations of devices, including those with Google Android systems, without the need for LiDAR sensors, using a camera to capture environment views, identifying feature points, generating meshes, and rendering AR objects to be occluded by real-world objects, while efficiently managing processing and battery power.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional object occlusion solutions are implemented, then occlusion functionality is achieved, but compatibility across mobile computing clients and operating systems is lost
Solution Approach 1:
The patent implements a universal point cloud processing framework that works across multiple mobile operating systems (iOS, Android, HarmonyOS) and device types. The system uses standardized APIs to capture point cloud data from diverse sources including LiDAR sensors, depth cameras, and monocular cameras, enabling object occlusion functionality to be universally applied across different mobile computing clients without requiring system-specific implementations
2Reliability
If conventional object occlusion solutions are implemented, then occlusion functionality is achieved, but processing power and memory requirements increase significantly
Solution Approach 1:
The patent segments the object occlusion process into distinct modular stages: point cloud capture, point cloud processing, mesh generation, and rendering. Each stage processes only the necessary data for that specific function, avoiding redundant computations. The system segments the environment into relevant objects and selectively processes only those areas requiring occlusion, significantly reducing overall processing power and memory consumption compared to conventional approaches that process entire scenes
Solution Approach 2:
The system performs partial processing by focusing computational resources only on regions where occlusion is actually needed rather than processing the entire environment. The point cloud processing selectively identifies and processes objects that will occlude AR content, leaving other areas unprocessed, thereby reducing energy consumption while maintaining occlusion functionality
3Measurement precision
If LiDAR sensors are used for object occlusion, then occlusion accuracy is improved, but device availability is reduced to only newer devices with LiDAR
Solution Approach 1:
The patent introduces multiple intermediary methods to bridge the gap between LiDAR-equipped devices and devices without LiDAR. The system uses depth cameras as an intermediary sensor type that can provide similar occlusion functionality through stereo vision or time-of-flight measurements. For devices without depth sensing capability, the system uses monocular camera-based depth estimation algorithms as intermediaries to infer 3D geometry from 2D images. These intermediary approaches enable occlusion accuracy close to LiDAR while expanding device availability to include older and more affordable devices
Solution Approach 2:
The system dynamically changes processing parameters based on the available sensor type and device capabilities. When LiDAR data is available, the system uses high-density point cloud processing parameters for maximum accuracy. When depth camera data is available, it adjusts parameters to optimize for stereo matching or ToF processing. When only monocular camera data is available, it uses parameters optimized for single-image depth estimation. This parameter adaptation maintains reasonable occlusion accuracy across different device generations and sensor types
Data Source
AI summary
An augmented reality (AR) game system hosted and executed on a mobile client enables occlusion of AR game elements by objects in the real world. The mobile client captures a camera view of an environment including a real-world object. The AR game system identifies a cluster of feature points associated with the real-world object and generates a mesh from the cluster. The AR game system represents the mesh on a virtual coordinate plane, such that an AR game element can appear to interact with the real-world object. The AR game system renders, on the mobile client, the AR game element to appear as though it is occluded by the real-world object.