In-App Communication Monitoring With ML Threat Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing in-app communication systems fail to effectively filter out threats such as bullying and sexual predation due to limitations in message filtering, particularly in understanding regional differences in slang and hidden meanings, and do not address auditory and video threats.
Innovation Solution
A child computing device captures screenshots, video clips, and audio samples, which are analyzed by a cloud-based machine learning model to identify potential threats, disabling the app if necessary and allowing parental override, with feedback to refine the model's understanding of in-app communication semantics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning is used to analyze in-app communications, then threat detection accuracy is improved, but device complexity increases
Solution Approach 1:
A cloud-based machine learning service is introduced as an intermediary between the child computing device and the threat detection system. The device captures screenshots, video clips, and audio samples, which are then transmitted to the cloud service for analysis. This intermediary handles the complex ML processing externally, improving detection accuracy without significantly increasing the complexity of the device itself.
Solution Approach 2:
The patent replaces traditional keyword-based filtering mechanisms with machine learning-based semantic analysis. Instead of relying on simple text matching algorithms, the system uses ML models that can understand context, slang, regional differences, and hidden meanings in communications, thereby improving detection accuracy through a fundamental substitution of the detection mechanism.
2Reliability
If comprehensive threat filtering is implemented, then safety is improved, but ease of operation deteriorates
Solution Approach 1:
The system incorporates feedback mechanisms where parents can review detected threats and provide feedback to the machine learning model. Parents can confirm false positives, mark false negatives, or provide contextual information about borderline cases. This feedback loop continuously improves the model's accuracy while maintaining parental control and ease of operation through a user-friendly interface.
Solution Approach 2:
The system performs preliminary filtering and analysis automatically before requiring parental intervention. The machine learning model pre-processes communications to identify potential threats, and only those that meet certain criteria are presented to parents for review. This preliminary action reduces the operational burden on parents while maintaining high safety standards.
3Adaptability or versatility
If multiple communication types are monitored, then threat detection coverage is improved, but device complexity increases
Solution Approach 1:
The system implements a universal monitoring framework that handles multiple communication types (text chat, voice messages, video calls, screenshots) through a single integrated machine learning service. The cloud-based platform processes all these different modalities using unified ML models, enabling comprehensive detection coverage without proportionally increasing device complexity, as the complexity is managed centrally in the cloud rather than distributed across the device.
Data Source
AI summary
A child computing device runs at least one app, such as a gaming app, a messaging app, social media app, and the like. A background application also runs on the child computing device and periodically takes samples of what the child is exposed to on the computing device, including samples of voice (which is converted to text), video streams (which are split into frames), and screenshots having text. A cloud computing platform provides services for machine learning (ML) to analyze the samples to ascertain a likelihood that the samples have threats (e.g., bullying or sexual predation). If it appears likely, the app is disabled and a notification is sent to a parental monitoring application along with a copy of the offensive sample. The parent can override the determination and the app is re-enabled. The parent's action is fed back to ML which learns from the feedback.