Tabular Data Compression for Anomaly Detection in Large Datasets

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Large datasets of user interactions, such as log records, are difficult to identify and analyze due to their size and complexity, making it challenging to detect distinct or anomalous interactions and determine the contributing attributes or events.

Innovation Solution

The system compresses a large data set into a representative data set while maintaining data density and variation, and generates an interface to automatically identify anomalies, cluster data, and visually represent the compressed data with emphasized anomalies, using machine-learning techniques and decision trees to select statistically important data fields for display in parallel coordinate charts.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of information

If the complete large data set is displayed, then data completeness is improved, but visualization complexity and analysis difficulty increase

Engineering Contradiction:

Improvedata completenessVSAvoidvisualization complexity

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The system creates a representative copy of the large dataset by selecting a smaller subset of rows that preserves the statistical properties, data density, and variation patterns of the original dataset. This representative sample is then displayed instead of the complete dataset, reducing visualization complexity while maintaining data completeness.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The system transforms the dataset by changing its parameters - specifically reducing the number of rows from the original large dataset to a optimized representative sample size. This parameter change maintains essential data characteristics while making the visualization manageable and analyzable.

Inventive Principle:

Principle #35Parameter changes

2Device complexity

If a smaller representative sample is displayed, then visualization complexity is reduced, but data representation accuracy may deteriorate

Engineering Contradiction:

Improvevisualization complexityVSAvoiddata representation accuracy

Core Design Contradiction:

Device complexityVSMeasurement precision

Solution Approach 1:

The system carefully adjusts the sampling parameters to optimize the balance between dataset size and representation accuracy. By using statistical methods to select the representative sample, the system ensures that the reduced dataset maintains accurate representations of data density, variation, and distribution patterns.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The system replaces manual sampling methods with automated machine-learning techniques and statistical algorithms that objectively select the most representative rows. This substitution ensures higher data representation accuracy compared to random or manual sampling approaches.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Loss of information

If all data fields are displayed, then information completeness is improved, but visual clarity deteriorates

Engineering Contradiction:

Improveinformation completenessVSAvoidvisual clarity

Core Design Contradiction:

Loss of informationVSEase of operation

Solution Approach 1:

The system extracts and displays only the most relevant and statistically important data fields from the complete set of columns. By identifying and removing less significant fields, the system maintains information completeness for key attributes while improving visual clarity by reducing the number of displayed columns.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The system applies different display qualities to different data fields based on their statistical importance and relevance. High-importance fields are displayed with full detail and emphasis, while less important fields are either summarized or excluded, creating a visually clear representation that maintains essential information.

Inventive Principle:

Principle #3Local quality

4Measurement precision

If manual analysis of large datasets is performed, then analysis thoroughness is improved, but time consumption increases

Engineering Contradiction:

Improveanalysis thoroughnessVSAvoidtime consumption

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system performs preliminary automated analysis of the entire large dataset before presenting the representative sample to the user. Statistical properties, data patterns, and anomalies are pre-calculated and embedded in the representative sample, allowing users to conduct thorough analysis quickly without manually examining the complete dataset.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system replaces manual data analysis with automated machine-learning algorithms and statistical computations. These automated systems rapidly process the complete dataset to identify patterns, anomalies, and key insights, then present them in the representative sample, dramatically reducing analysis time while maintaining thoroughness.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS11354836B2Systems and methods for displaying representative samples of tabular data

Publication Date: 2022.06.07 ORACLE INT CORP

AI summary

The present disclosure generally relates to systems and methods that efficiently display tabular data (e.g., a large data set of a million or more rows of data with multiple data fields). More particularly, the present disclosure relates to systems and methods that compress the tabular data to a representative data set that maintains the data density and data variation of the original tabular data, and that display the representative data set with respect to clusters formed.