Value Record Indexing for Schema-Independent Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data analytics systems struggle to efficiently analyze and process data values without relying on the entire record structure, especially when dealing with unknown or changing data formats.
Innovation Solution
The system breaks down records into value records, extracts metadata, and indexes these value records based on their data values. It then uses value pattern instances to establish relationships between value records, producing value pattern outcomes without moving the entire record.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is analyzed as complete records with all fields moved to perform processing, then processing can be done with full context, but processing efficiency decreases and flexibility to handle unknown schemas is reduced
Solution Approach 1:
The patent segments complete records into individual field-value pairs, allowing each value to be processed independently rather than moving entire records. This segmentation enables selective processing of only the necessary data elements, improving efficiency while maintaining flexibility to handle varying schemas.
Solution Approach 2:
The patent extracts specific field values from records and processes them separately from the rest of the record structure. By taking out only the necessary values for analysis and leaving the complete record structure intact, the system achieves both processing efficiency and schema flexibility.
2Productivity
If specific fields must be known and specified to perform processing, then processing can be targeted and efficient, but the system cannot handle unknown or changing data formats
Solution Approach 1:
The patent creates a universal processing framework that can handle any field value regardless of its specific type or origin. By processing data as generic values with associated metadata rather than requiring specific field knowledge, the system achieves both targeted efficiency and universality across unknown schemas.
Solution Approach 2:
The patent changes the processing parameter from field-specific processing to value-based processing. Instead of requiring knowledge of specific field names and structures, the system processes based on value characteristics and relationships, enabling adaptation to unknown formats while maintaining processing efficiency.
3Loss of information
If all fields about an entity are moved from place to place for processing, then complete entity context is maintained, but processing time and resource consumption increase
Solution Approach 1:
The patent extracts only the specific field values needed for processing while leaving the complete record structure in place. This extraction approach maintains entity context through value relationships and metadata while avoiding the time-consuming operation of moving entire records.
Solution Approach 2:
The patent segments the data processing into value extraction and value processing phases, allowing entity context to be preserved in the original records while processing operates efficiently on extracted values. This segmentation eliminates the need to move complete records while maintaining contextual integrity.
Data Source
AI summary
Methods and systems for analyzing data values are disclosed. From one or more sources, source data is received as records, with each record including data values. For each record, the record is broken into value records, and from the record metadata about the record is extracted, where each value record includes one data value. The value records are indexed based on the respective data values of the value records, to produce value indices. One or more value pattern instances of a value pattern is/are used to establish relationships between the value records and produce value pattern outcomes based on at least one of: the value indices or the respective data values of the value records.