Data Refinement Engine Distributed Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computer systems face performance degradation due to the significant difference in data retrieval times between Random Access Memory (RAM) and disk storage, leading to inefficient data processing and analysis.
Innovation Solution
An in-memory engine collaborates with a big data engine to process and refine data, distributing processing tasks between the two systems to balance loads and improve data retrieval efficiency, utilizing techniques like data wrangling, filtering, and analytics to enhance data processing capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in disk storage, then storage capacity is increased, but data retrieval time increases significantly
Solution Approach 1:
The patent segments data storage into two distinct layers: disk storage for bulk data retention and RAM for active data processing. The data refinement engine selectively loads only the necessary subsets of data from disk into RAM based on query requirements, rather than loading entire datasets. This segmentation allows the system to maintain large storage capacity while minimizing data retrieval time by working with smaller, pre-loaded data subsets in memory.
2Manufacturing precision
If all data processing is performed by the data refinement engine, then data processing quality is improved, but processing load and time increase
Solution Approach 1:
The patent implements preliminary action by having the data refinement engine pre-process and refine data subsets before they are loaded into RAM. The engine performs filtering, aggregation, and other data refinement operations on disk-stored data before transfer to memory, so that when data is loaded, it is already optimized for the specific query. This eliminates the need to load and process entire datasets, significantly reducing processing time while maintaining data quality.
3Speed
If data is loaded entirely into RAM, then data access speed is improved, but memory usage and cost increase
Solution Approach 1:
The patent applies partial action by loading only the specific subsets of data that are currently needed for active queries into RAM, rather than loading entire datasets. The data refinement engine determines which data subsets will be required based on query analysis, and loads only those portions into memory. This approach maintains fast data access speeds for active data while keeping memory usage proportional to actual needs rather than total data volume.
Data Source
AI summary
Data engine integration and data refinement are described. The actions include receiving, by a data refinement engine, a request for data. The actions include determining a first amount of processing to be performed by the data refinement engine and a second amount of processing to be performed by one or more processors of a data source that include a plurality of data nodes. The actions include transmitting, by the data refinement engine, code to the plurality of data nodes of instructions associated with the second amount of processing. The actions include receiving, by the data refinement engine and from the plurality of data nodes, unprocessed first data and processed second data. The actions include processing, by the data refinement engine, the unprocessed first data. The actions include, in response to the request for data, transmitting, by the data refinement engine, the processed first data and the processed second data.