Data Loading Indexing via Preliminary Column Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing business systems face challenges in activating the online system on time due to increased data storage and index creation times when the number of users exceeds estimates, potentially leading to delayed service activation.
Innovation Solution
The system prioritizes the acquisition and storage of data corresponding to columns used for index creation, allowing for index creation to start before all data is stored, thereby reducing the overall data storage and index creation time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in sequential order from first column to last column, then data storage completeness is ensured, but activation time is delayed
Solution Approach 1:
The system performs preliminary actions by storing data from columns that will be used for index creation before storing data from other columns. This allows index creation to begin earlier in the process, reducing the overall activation time while still ensuring complete data storage eventually occurs.
Solution Approach 2:
The data storage process is segmented into different phases: first storing data from columns needed for index creation, then storing data from remaining columns. This segmentation allows index creation to proceed in parallel with the second phase of data storage, resolving the contradiction between completeness and timing.
2Manufacturing precision
If all data is stored before index creation, then data accuracy for indexing is ensured, but overall processing time increases
Solution Approach 1:
The system performs preliminary storage of data from columns that will be used for index creation before initiating the indexing process. This preliminary action ensures that accurate data is available for indexing without requiring complete storage of all data columns first, thereby improving processing speed while maintaining data accuracy for the indexing operation.
3Reliability
If the system waits for complete data storage before activation, then system reliability is improved, but service activation is delayed
Solution Approach 1:
The system performs preliminary storage of critical data columns and preliminary index creation before full data storage is complete. This allows the online system to be activated earlier with sufficient data and indexes for core operations, improving service activation time while maintaining adequate system reliability for essential functions.
Solution Approach 2:
The system dynamically adjusts the activation timing based on the completion status of critical data storage and index creation. Rather than waiting for complete data storage, the system can activate when critical thresholds are met, balancing reliability requirements with activation timing needs.
Data Source
AI summary
An apparatus for data loading includes a memory and a processor coupled to the memory and the processor configured to, execute first acquisition of first data corresponding to a first column from a data group stored in a first storage, a data structure of the data group including columns and rows, store the first data into a second storage, and generate a first index based on the first data for searching the data group to be stored in the second storage in response to the first acquisition before completion of storing the data group into the second storage.