Tabular Data Location via Statistical Synopses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for locating tabular data across different storage systems are inefficient due to their reliance on bit-level fingerprinting, which is not robust against modifications, and watermarking, which is impractical for large-scale datasets across various storage systems.
Innovation Solution
A computer-implemented method that computes statistical synopses for columns of tabular data, allowing for the comparison of subsets of descriptors to identify potential matches between datasets stored in different systems, reducing computational effort and storage requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If bit-level fingerprinting is used to identify tabular data, then data identification accuracy is improved, but robustness against modifications deteriorates
Solution Approach 1:
The patent segments the tabular data into column-level units and computes statistical synopses for each column independently. This segmentation allows the system to identify data matches at the column level rather than requiring exact bit-level matches of entire datasets, thereby improving robustness against modifications while maintaining identification accuracy through statistical comparison of column descriptors.
Solution Approach 2:
The patent transforms the data representation from bit-level fingerprints to statistical parameters (synopses) that capture the essential characteristics of columns. By changing the parameter space from exact bit patterns to statistical measures, the system becomes robust against modifications while maintaining the ability to identify matching data through parameter comparison.
2Loss of information
If watermarking is applied to all rectangular datasets, then data traceability is improved, but system complexity and implementation difficulty increase
Solution Approach 1:
The patent enables the data identification system to automatically compute statistical synopses and perform matching without requiring external watermarking infrastructure. The system serves itself by using the inherent statistical properties of the data columns to create identifiers, eliminating the need for complex watermarking algorithms and reducing system complexity while maintaining traceability.
Solution Approach 2:
The patent introduces statistical synopses as an intermediary representation between the raw tabular data and the identification process. This intermediary captures the essential characteristics of data columns without requiring direct modification of the data or complex watermarking, thereby improving traceability while reducing system complexity.
3Measurement precision
If comprehensive data fingerprinting is performed across all storage systems, then data location accuracy is improved, but computational effort and storage requirements increase
Solution Approach 1:
The patent extracts only the essential statistical characteristics (synopses) from the tabular data columns rather than processing the entire datasets. This extraction approach maintains data location accuracy by preserving the key identifying features while dramatically reducing computational effort and storage requirements compared to comprehensive fingerprinting of all data.
Solution Approach 2:
The patent applies partial action by computing synopses for only the necessary column subsets and comparing only relevant descriptors. This approach achieves sufficient data location accuracy without the excessive computational burden of processing all possible data attributes, thereby improving productivity while maintaining measurement precision.
Data Source
AI summary
An approach to finding data locations may be provided. A first synopses, which corresponds to first tabular data may be provided. An auxiliary data storage system may be scanned to identify second tabular data stored in the auxiliary data storage system. A second synopses may be obtained, in which the synopses correspond to the second columns of second tabular data. The synopsis may be computed for each second column of the second columns, according to a numeric representation of contents of cells of said each second column. The computed synopsis may include a vector of m descriptors. The two sets of one or more descriptors may be compared. A subset of the descriptors of the second synopses may be compared with corresponding descriptors of the first synopses, to identify potential matches between the second tabular data and the corpus of first tabular data.