Big Object Memory-Mapped Data Structure for Real-Time Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current relational databases and OLAP technologies are inefficient in processing large volumes of data, leading to significant processing times that hinder real-time decision-making for business managers.
Innovation Solution
An apparatus and method that utilize memory mapped files and Location-Independent Structure (LIS) to create a Big Object from big data, organizing its content into meta information, tree, and data sections, allowing for efficient access and processing by arranging measures in a continuous memory space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional relational databases and OLAP technologies are used to store and process big data, then data storage and processing capabilities are provided, but processing time becomes significantly long when dealing with huge amounts of data
Solution Approach 1:
The patent segments big data into multiple partitions and distributes them across different storage nodes. Each partition can be processed independently and in parallel, transforming a single long-processing task into multiple shorter parallel tasks, thereby reducing overall processing time while maintaining data storage capability
Solution Approach 2:
The patent introduces a new dimensional organization for data storage by creating a multi-dimensional data model that extends beyond traditional relational database structures. This allows data to be accessed and processed from multiple dimensions simultaneously, enabling parallel processing operations that reduce processing time while maintaining comprehensive data storage
2Ease of operation
If data is stored in traditional database structures, then systematic data organization is achieved, but real-time analytical processing becomes impossible due to long access and analysis times
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing data in multiple pre-processed formats and dimensions. When a query is executed, the system can directly retrieve pre-computed results without performing complex real-time calculations, enabling real-time analytical processing while maintaining systematic data organization
Solution Approach 2:
The patent introduces an intermediary layer between the stored data and the query processing system. This intermediary layer maintains multiple indexed views and pre-computed aggregates of the data, allowing rapid query resolution without directly accessing the base data structures, thereby enabling real-time access while preserving systematic organization
Data Source
AI summary
An apparatus and method for realizing big data into a Big Object and a non-transitory tangible machine-readable medium are provided. The apparatus comprises an interface and a processor. The interface is configured to access big data stored in a storage device. The processor is configured to create the Big Object from the big data using memory mapped files. The processor further lays out a content of the Big Object, wherein the content comprises a meta information section, a tree section, and a data section. The processor further lays out a content of the tree section by using LIS and describes a structure of the Big Object in the meta information section.