Data Intake Query System Record Expansion Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data intake and query systems face challenges in seamlessly searching and analyzing large volumes of diverse data from various sources, including structured, semi-structured, and unstructured data, due to limited scope and unidirectional processing flows that restrict access to external data systems.
Innovation Solution
A data intake and query system is developed with a network of distributed nodes and a search process master that extends search and analytics capabilities across diverse data systems, enabling scalable processing and integration with external data sources, such as MySQL, PostgreSQL, and cloud storage, to provide comprehensive search and analysis capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored in a unified data lake format for flexible analysis, then adaptability and analytical flexibility improve, but data retrieval speed and query performance deteriorate due to lack of structured organization
Solution Approach 1:
The patent segments data into two distinct storage formats: a data lake for raw, flexible storage and a data warehouse for structured, optimized query performance. This segmentation allows different data access patterns to be served by appropriate storage systems, resolving the contradiction between flexibility and speed.
Solution Approach 2:
The patent introduces an intermediary data warehouse layer between the data lake and query systems. This intermediary pre-processes and structures data from the data lake, enabling fast retrieval while maintaining the flexibility of raw data storage in the data lake.
2Productivity
If distributed processing nodes are added to handle large data volumes, then processing capacity and scalability improve, but system complexity and coordination overhead worsen
Solution Approach 1:
The patent segments the distributed system into specialized nodes with distinct roles: data lake nodes for flexible storage, data warehouse nodes for structured storage and processing, and query nodes for handling requests. This role-based segmentation manages complexity by creating clear boundaries and responsibilities.
Solution Approach 2:
The patent creates multi-functional distributed nodes that can operate in different modes. Nodes can dynamically switch between data lake and data warehouse operations, reducing overall system complexity by reusing infrastructure rather than requiring separate specialized systems.
3Loss of time
If data is pre-processed and filtered before storage to reduce volume, then storage costs and query time improve, but loss of raw data information and analytical flexibility worsen
Solution Approach 1:
The patent segments data storage into raw data lake (preserving all information) and processed data warehouse (optimized for queries). This allows simultaneous retention of complete raw data for future analysis and pre-processed data for current query performance.
Solution Approach 2:
The patent performs preliminary processing actions by creating structured copies of data in the data warehouse before queries are executed. This preliminary structuring enables fast query performance without losing the ability to analyze original raw data in the data lake.
Data Source
AI summary
Systems and methods are described for processing records associated with a query that identifies an association between two data fields. The system can obtain a chunk of data that includes multiple records based on a query received by a data intake and query system. At least one record can include multiple sub-records that share a field value for at least one field. The system can generate a record from each sub-record and assign the generated records to one or more groups of partitions. The system can combine record data of generated records assigned to one partition of a group of partitions and then combine record data across the group of partitions. The system can process the results of the combination of records across the group of partitions based on the query.