Cloud Data Investigation System for High-Dimensional Integrity Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing methods are inefficient for rapid data investigation and integrity analysis, particularly in high-dimensional data sets, where visualization and pattern detection are computationally intensive and difficult, leading to challenges in data understanding and predictive analytics.
Innovation Solution
A cloud-based system and method for rapid data investigation and integrity analysis, enabling multiple users to collaborate and generate graphical representations of statistics, with distributed storage and parallel processing for efficient data exploration and validation, ensuring data adherence to formats and behaviors during model deployment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multivariate binning is used for high-dimensional data analysis, then detailed distribution estimates can be obtained, but the number of bins becomes intractable and computation becomes too intensive
Solution Approach 1:
The patent segments the high-dimensional data space by identifying and analyzing individual data elements separately rather than creating a complete multivariate binning structure. This avoids the combinatorial explosion of bins while still enabling detailed analysis of each dimension's distribution and patterns.
Solution Approach 2:
The patent extracts and analyzes individual data elements and their distributions separately from the full multivariate space. By taking out each element's distribution for separate analysis, the system avoids the need to construct and manage the complete intractable binning structure while still obtaining detailed distribution estimates.
2Reliability
If comprehensive data understanding and integrity analysis is performed, then data quality for predictive analytics is improved, but the process becomes time-consuming and slower than model development
Solution Approach 1:
The patent performs preliminary analysis by automatically generating summaries, distributions, and pattern detections for all data elements before the data scientist begins model development. This preliminary action captures critical data understanding information in advance, reducing the time needed during the actual modeling phase while maintaining comprehensive data quality assessment.
Solution Approach 2:
The system performs self-service data analysis by automatically computing distributions, detecting patterns, identifying anomalies, and generating insights without requiring manual intervention from the data scientist. This automation maintains comprehensive data understanding while dramatically improving analysis speed and productivity.
3Loss of information
If manual data inspection and analysis are performed by data scientists, then detailed understanding of data patterns and integrity can be achieved, but the process is time-consuming and may miss automatic patterns
Solution Approach 1:
The patent implements feedback by having the system automatically analyze data and provide insights, patterns, and anomalies back to the data scientist. This feedback loop enables comprehensive pattern detection that would be difficult to achieve manually while significantly reducing the time investment required from the data scientist.
Solution Approach 2:
The patent replaces the mechanical manual inspection process with automated computational analysis. The system uses algorithms to detect patterns, analyze distributions, and identify data integrity issues, substituting human manual analysis with automated systems that are both faster and more comprehensive in detecting patterns.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system and method for rapid data investigation and data integrity analysis is disclosed. A data set is received by a server computer from one or more client computers connected with the server computer via a communications network, and the data set is stored in a distributed storage memory. One or more analytical processes are executed on the data set from the distributed storage memory to generate statistics based on each of the analytical processes, and the statistics are stored in a random access memory, the random access memory being accessible by one or more compute nodes, which generate a graphical representation of at least some statistics stored in the random access memory. The graphical representation of at least some statistics is then formatted for transmission to and display by the one or more client computers.