Big Data Query Algorithm Formation via Dependency Graph Reverse Engineering

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data retrieval methods face challenges in efficiently analyzing large quantities of semi-structured and structured data, known as 'big data,' due to its heterogeneous nature and the difficulty in navigating and organizing it for trend analysis, often requiring time-consuming and iterative programming approaches.

Innovation Solution

A method for creating and executing query algorithms that involves loading a sample population of records into a database cluster, using globally distributed data structures and fat pointers to optimize database operations such as FILTERS, AGGREGATES, and JOINS, allowing for interactive exploration and reverse engineering of database operations to identify useful schemas, and dynamically managing memory and indexing for efficient data retrieval.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If data is stored in heterogeneous semi-structured formats across multiple files, then data volume and flexibility are increased, but navigation and organization difficulty increase

Engineering Contradiction:

Improvedata volumeVSAvoidnavigation difficulty

Core Design Contradiction:

Quantity of substanceVSEase of operation

Solution Approach 1:

The patent segments the heterogeneous big data into structured tables with standardized schemas. Each semi-structured file is parsed and its content is organized into discrete table rows with consistent column definitions, enabling systematic navigation while preserving the original data volume and diversity

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary layer consisting of standardized table schemas that mediate between the heterogeneous source files and the query processing system. This intermediary structure provides a unified interface for navigation while accommodating diverse data formats underneath

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If data is organized into structured tables with specific schemas, then query efficiency is improved, but adaptability to heterogeneous data formats deteriorates

Engineering Contradiction:

Improvequery efficiencyVSAvoidformat flexibility

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The patent implements dynamic schema evolution capabilities that allow table structures to adapt to new data formats. The system can automatically detect new field types and schemas from incoming heterogeneous data and update the structured table definitions accordingly, maintaining query efficiency while accepting format diversity

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent changes the schema parameters of structured tables to accommodate different data formats. By allowing schema definitions to be modified based on the characteristics of incoming semi-structured data, the system maintains efficient structured storage while adapting to format variations

Inventive Principle:

Principle #35Parameter changes

3Ease of manufacture

If traditional shot-in-the-dark approach is used for data analysis, then programming simplicity is maintained, but analysis time and resource consumption increase

Engineering Contradiction:

Improveprogramming simplicityVSAvoidanalysis time

Core Design Contradiction:

Ease of manufactureVSLoss of time

Solution Approach 1:

The patent performs preliminary actions by automatically generating and executing candidate queries based on the structured data organization. The system pre-processes the data into queryable formats and uses automated algorithms to generate relevant queries, eliminating the need for manual trial-and-error programming while reducing analysis time

Inventive Principle:

Principle #10Preliminary action

4Measurement precision

If multiple iterations of programming are performed to find useful trends, then data exploration thoroughness is improved, but development time and complexity increase

Engineering Contradiction:

Improvetrend detection accuracyVSAvoidprogramming complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent implements self-service functionality where the system automatically performs trend detection and analysis without requiring multiple manual programming iterations. The automated query generation and execution system continuously analyzes the structured data and identifies trends, providing thorough exploration while maintaining simple operation for the user

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS9805080B2Data driven relational algorithm formation for execution against big data

Publication Date: 2017.10.31 XCALAR

AI summary

Techniques are described herein for creating an algorithm for batch mode processing against big data. The techniques involve receiving one or more user commands from a set number of commands that correspond one-to-one with a set number of low-level database operations. In a preferred embodiment, the set of database operations includes only FILTERS, SORTS, AGREGGATES, and JOINS.In the algorithm formation process, database operations are performed on a sample population of records. The user drills down to a set of useful records by performing database operations against the results of the previous database operations. While the database cluster is receiving operations, the system is tracking the operations in a dependency graph. The chains selected within the dependency graph indicate which operations are used to create the algorithm. To generate the algorithm, the database cluster reverse engineers the logic for performing those operations against big data.