Local Data Container Segmentation for High-Speed Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing third-party database management systems struggle to fully satisfy specific data processing requirements, particularly in terms of high-speed data processing and efficiency, due to communication bottlenecks and resource consumption when handling massive data.
Innovation Solution
A method and system that divide a data table into a preset number of data pages or containers with pagination indexes, allowing each container to perform direct data processing in response to instructions, eliminating the need for a third-party database server and reducing communication overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a third-party database management system is used to process data, then data processing can be performed with mature and versatile functionality, but communication efficiency becomes a bottleneck and large amounts of memory are consumed when handling massive data
Solution Approach 1:
The patent extracts the data storage and processing functionality from the third-party database server and implements it directly within the client application through local data pages. This eliminates the need for remote communication with a database server, thereby removing the communication efficiency bottleneck while maintaining full data processing capabilities locally.
Solution Approach 2:
The patent segments the data table into multiple data pages, each stored locally in the client application. This segmentation allows the client to process data directly without relying on a centralized server, improving productivity while maintaining versatile data processing functionality through the distributed data page structure.
2Speed
If a service-type memory database is used for high-speed data read and write, then data read and write efficiency is improved, but communication efficiency becomes a bottleneck and large amounts of memory are consumed
Solution Approach 1:
The patent extracts the data storage functionality from the service-type database and implements local data pages within the client application. This eliminates remote communication overhead entirely, achieving high-speed data processing without the communication efficiency bottlenecks associated with service-type databases.
Solution Approach 2:
The client application performs data storage and processing operations directly using local data pages without requiring service-type database interactions. This self-service approach eliminates communication dependencies, enabling high-speed data read and write operations while improving overall communication efficiency.
3Adaptability or versatility
If a third-party database server is deployed for data processing, then data can be processed with mature functionality, but communication overhead increases and processing efficiency decreases
Solution Approach 1:
The patent extracts data processing capability from the remote server and implements it locally through data pages stored in the client application. This eliminates all communication time associated with server interactions, enabling direct data processing while maintaining full data processing capability through the local data page structure.
Solution Approach 2:
By segmenting data into local data pages within the client application, the patent enables processing operations to be performed locally without time-consuming communication to a remote server. This segmentation strategy eliminates communication time loss while preserving comprehensive data processing capability.
Data Source
AI summary
A method, a system and an apparatus are provided for processing massive data at a high speed, and a storage medium, which relate to the technical field of data processing. A data table where a target data type is stored is divided into a plurality of containers, where the containers store data and have pagination indexes, and each container is directly controlled to perform corresponding data processing according to the pagination index of each container in response to a data processing instruction. According to the solution, data processing can be directly performed by means of the data containers obtained by dividing, no third-party database server is required to be deployed, such that the solution is more advantageous in client application; and without reliance on service, performance consumption of communication with a service is reduced, and data processing efficiency is higher. The data table is divided into the plurality of containers, such that data processing can be performed by utilizing multi-threads, and data processing efficiency is improved.