Data Import System Bulk Loading and Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for importing external data into systems are inconvenient and time-consuming, as they require extensive processing and handling of multiple data files.
Innovation Solution
A method and device for importing multiple data sets into a system, where data sets are read from a storage medium, stored without processing in a first table, and then written into sub-tables with predefined categories, allowing for efficient data mapping, checking, and error tracking, using a separator like commas or tabs to separate data segments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data sets are imported using traditional file handling methods, then data can be imported into the system, but the process is inconvenient and time-consuming
Solution Approach 1:
The patent segments the data import process into distinct phases: initial bulk loading of raw data sets into a first table without processing, followed by subsequent processing steps that divide data into data segments and write them to sub-tables. This segmentation allows the system to handle large volumes of data efficiently while maintaining the ability to process and validate data in manageable portions, thereby reducing overall import time and improving convenience.
Solution Approach 2:
The patent applies preliminary action by loading all data sets into the first table in advance without immediate processing. This preliminary bulk load enables subsequent processing operations to work with pre-loaded data, eliminating the need for repeated reading from storage media and significantly reducing the time required for actual data processing and validation steps.
2Reliability
If data sets are processed extensively during import, then data integrity can be ensured, but processing time increases
Solution Approach 1:
The patent segments data processing operations into distinct phases: initial bulk loading without processing, followed by subsequent processing that divides data into segments with specific categories. This allows the system to maintain data integrity through structured processing while improving productivity by avoiding redundant operations on already-processed data segments.
Solution Approach 2:
The patent applies local quality by creating sub-tables with specific data fields and categories tailored to different types of data segments. Each sub-table is optimized for its specific purpose, allowing the system to ensure data integrity for each category while maintaining overall high-speed import capability through targeted rather than universal processing.
3Adaptability or versatility
If multiple data files are used for data import, then comprehensive data can be imported, but file handling becomes complex and time-consuming
Solution Approach 1:
The patent merges multiple data sets from different files into a single first table during the initial bulk load phase. This consolidation eliminates the need for complex file handling operations during subsequent processing, as all data is already available in one location. The system maintains adaptability to import various data types while simplifying the overall process by combining file handling into a single initial operation.
Data Source
AI summary
A method for importing a plurality of data sets into a system, each data set comprising data segments separated by a separator, the method comprising the steps of: providing data sets to be imported on a data storage medium, reading the data sets from the data storage medium, storing the data sets in a first table in the system, substantially without processing the data sets, reading the stored data sets from the first table, and writing one or more data segments of each data set into sub-tables, each sub-table comprising at least one data field. Furthermore, it is disclosed a computer program product and a device for importing a plurality of data sets into a system.