Vector Projection for Reducing Data Density in Large Datasets

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Managing and searching large datasets is challenging due to high data density, particularly in systems that require identifying closest matches between data points, where storage of actual data points occupies significant database volume.

Innovation Solution

The system projects vectors from an origin point to determine the number of vectors between a reference data point and an unknown data point, estimating the angle and distance between them, allowing for the identification of candidate data points without storing actual data points, thereby reducing data density and improving search efficiency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If actual data points are stored in large datasets, then complete information is available for accurate matching, but storage volume and data density increase significantly

Engineering Contradiction:

Improvematching accuracyVSAvoiddata volume

Core Design Contradiction:

Measurement precisionVSQuantity of substance

Solution Approach 1:

The patent extracts only the essential geometric properties (vector lengths and angular relationships) from the complete data points, storing only these extracted features rather than the full data. This allows accurate distance calculation through geometric reconstruction without storing redundant information, resolving the contradiction between storage efficiency and matching accuracy.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent transforms the data representation from storing complete data points to storing transformed parameters (vector lengths and projected vector counts). This parameter transformation enables the system to reconstruct distances using geometric relationships rather than storing all original data, achieving both compression and accuracy.

Inventive Principle:

Principle #35Parameter changes

2Reliability

If all reference data points are stored for searching, then complete candidate identification is possible, but search and management complexity increases

Engineering Contradiction:

Improvecandidate identification completenessVSAvoidsearch system complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent segments the distance calculation process into two independent components: vector length (magnitude) and angular relationship (direction). By storing these segmented components separately and combining them through geometric formulas, the system simplifies the search structure while maintaining complete candidate identification capability.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces projected vectors as intermediary elements that mediate between stored reference data and query data. These projected vectors serve as a computational bridge, enabling distance calculation without direct access to complete reference data points, thus reducing search complexity while preserving identification accuracy.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Quantity of substance

If vector projection method is used to estimate angles and distances, then data storage is reduced, but calculation complexity increases

Engineering Contradiction:

Improvestored data amountVSAvoidcalculation process complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The patent performs preliminary projection of reference vectors onto a set of fixed projection directions during data preparation. This preliminary action pre-computes the angular relationships in a simplified coordinate system, transforming complex multi-dimensional angle calculations into simpler one-dimensional projected vector comparisons, thereby reducing online calculation complexity.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent substitutes direct geometric distance calculation (which would require complete data points) with an alternative computational mechanism based on projected vector relationships. This mechanism uses algebraic operations on projected components rather than direct geometric computation, simplifying the calculation process while maintaining accuracy.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS10192138B2Systems and methods for reducing data density in large datasets

Publication Date: 2019.01.29 INSCAPE DATA INC

AI summary

Techniques and systems are provided for identifying unknown content. For example, a number of vectors out of a plurality of vectors projected from an origin point can be determined that are between a reference data point and an unknown data point. The number of vectors can be used to estimate an angle between a first vector (from the origin point to a reference data point) and a second vector (from the origin point to an unknown data point). A distance between the reference data point and the unknown data point can then be determined. Using the determined distance, candidate data points can be determined from a set of reference data points. The candidate data points can be analyzed to identify the unknown data point.