Pixel-Based Game Testing Using Object Detection and Exploration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Testing of complex virtual environments in computer programs, such as virtual reality and computer games, is becoming increasingly costly and time-consuming due to their size and complexity, and existing automated testing methods require access to source code, limiting their applicability and flexibility.
Innovation Solution
A computing system utilizing a machine learning model with integrated object detection, investigation, and environment exploration modules, which processes screen images to detect and interact with key objects and explore environments independently of the application's internal state, enabling thorough testing without source code access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If human testers and human-written scripts are used for testing, then testing thoroughness can be maintained, but testing time and cost increase significantly as program size and complexity grow
Solution Approach 1:
The testing system performs self-service by using machine learning models to automatically detect objects, generate test inputs, and explore environments without human intervention. The system trains itself on game data and autonomously executes testing tasks, eliminating the need for human testers to manually perform repetitive testing actions while maintaining comprehensive test coverage
Solution Approach 2:
The patent replaces the mechanical system of human testers physically interacting with games by substituting it with an automated ML-based testing agent. This agent uses computer vision to perceive game states, processes game logic through neural networks, and generates test inputs automatically, thereby reducing testing time while maintaining thoroughness through consistent and exhaustive automated execution
2Productivity
If traditional automated testing methods requiring source code access are used, then testing efficiency improves, but applicability is limited to applications where source code can be accessed
Solution Approach 1:
The patent introduces an intermediary approach by using screen capture images as a mediator between the testing system and the application under test. Instead of directly accessing source code, the ML model observes the graphical interface and infers application state, enabling testing of third-party applications where source code is inaccessible while maintaining efficient automated testing through image-based state recognition
Solution Approach 2:
The system creates a virtual copy of the application interface through screen captures and uses this copy for testing purposes. The ML model processes these visual copies to understand application state and generate test inputs, allowing the testing system to interact with applications as if it were a user, thereby expanding applicability to any application with a graphical interface regardless of source code accessibility
3Reliability
If extensive manual testing is performed to ensure quality, then bug detection improves, but development cost and time consumption increase
Solution Approach 1:
The patent applies preliminary action by training the machine learning model in advance using game data and object detection algorithms. This pre-trained model is then deployed for automated testing, eliminating the need for costly and time-consuming manual testing during development. The preliminary training phase invests computational resources upfront, enabling efficient and scalable bug detection throughout the development lifecycle without increasing ongoing development costs
Data Source
AI summary
A computing system for pixel-based automated testing of a navigable simulated environment includes one or more processors configured to execute, in a run-time inference phase, an application testing program. The application testing program includes a machine learning model trained to detect errors in the application under test. A screen image of the simulated environment is processed by an object detection module to determine if a key object is present in the screen image. If a key object is present in the screen image, the application testing program executes an object investigation module to generate investigation inputs to investigate the key object. If a key object is not present in the screen image, the application testing program executes an environment exploration module to generate an environment exploration action to be provided to the application under test as simulated user input.