AI Anomaly Search in Electronic Records Using Clustered Ensemble Checks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data search technologies face challenges in scalability, data source integration, and accuracy, particularly in financial instrument trading, due to the need for customized tools and the issue of 'hallucination' in large language model (LLM) responses, which can violate rules and guidelines.
Innovation Solution
An AI-based system that integrates large language models with machine learning agents for orchestration, enabling scalable data search and analytics, while ensuring compliance and reducing hallucination through parallel checks and verifications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing data search technologies are used, then data search functionality is provided, but scalability is limited and infrastructure costs are high
Solution Approach 1:
The patent implements a unified search architecture that handles multiple data types (structured, unstructured, semi-structured data) and multiple search modalities (keyword search, semantic search, vector search) through a single system. This multi-functional design eliminates the need for separate customized tools for different data sources, achieving scalability while controlling infrastructure costs through resource sharing and consolidation.
2Ease of operation
If large language models are used for data search, then natural language querying is enabled, but hallucination occurs violating rules and guidelines
Solution Approach 1:
The patent introduces an intermediary verification layer between the large language model and the final search results. This intermediary component validates LLM outputs against ground truth data, cross-checks responses for hallucinations, and ensures compliance with rules and guidelines. The intermediary acts as a mediator that preserves the ease of natural language querying while filtering out unreliable or hallucinated content before presentation to users.
3Adaptability or versatility
If customized tools are used for each data source, then data source integration is achieved, but device complexity and costs increase
Solution Approach 1:
The patent implements a universal data access layer that provides standardized interfaces for connecting to diverse data sources including databases, data lakes, and external APIs. This unified approach allows the system to integrate multiple data sources without requiring customized tools for each source, reducing overall system complexity while maintaining adaptability to different data formats and storage mechanisms through standardized connection protocols.
Data Source
AI summary
In order to facilitate artificial intelligence-based anomaly detection in electronic records, systems and methods include establishing a computer-implemented clustering process based on record attributes (e.g., using a k-means algorithm) such that records are grouped into clusters and pairs of records within each cluster are subsequently analyzed; generating feature vectors by normalizing and concatenating selected attributes and filtering out vectors exhibiting low variance based on first predetermined parameters; applying an ensemble of at least three anomaly detection models (e.g., Local Outlier Factor, DBSCAN and another model) to cast votes on whether each pair is anomalous and flagging pairs that satisfy a second predetermined consensus threshold; and performing actions in response to flagged anomalies, including generating alerts or storing results with associated anomaly scores, whereby the system enhances detection accuracy and scalability for applications such as financial market analysis and other domains requiring robust data evaluation.