Database Data Import Config File Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems for transportation big data reports are inefficient, with a single data form and fixed request parameters that cannot meet diverse data acquisition needs, leading to low scalability and increased complexity in data processing and integration.
Innovation Solution
A method of importing data to a database that involves acquiring original data from sources, processing it according to a pre-configured database config file to generate computational data, and writing both types of data into the database, with features like data slicing and a staging area to enhance efficiency and security.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is acquired by traversing and accessing database interface one data per access, then complete data can be obtained for big data report, but data acquisition efficiency is very low and process complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-configuring database connection information, table structures, and field mappings in a configuration file before data acquisition. This allows the system to efficiently retrieve and process multiple data fields in batch operations rather than accessing the database interface one data point at a time, thus improving acquisition efficiency while maintaining data completeness
2Ease of manufacture
If data acquisition method uses fixed request parameters, then system implementation is simple, but it cannot meet diverse data acquisition needs and scalability is reduced
Solution Approach 1:
The patent implements dynamics by designing a configurable data acquisition system where request parameters, database connection information, and processing rules can be dynamically adjusted through configuration files. This allows the system to adapt to diverse data acquisition needs without complex reprogramming, balancing implementation simplicity with operational versatility
Solution Approach 2:
The patent applies universality by creating a unified data acquisition framework that can handle multiple data sources, different database types, and various processing requirements through a single configurable system. The configuration file approach allows one system to serve multiple functions for different data acquisition scenarios
3Reliability
If all incoming data and computational data are written to database, then data availability for analysis is improved, but database system burden increases
Solution Approach 1:
The patent applies extraction by separating computational data processing from incoming data storage. Computational data is calculated and processed separately based on incoming data, then both types of data are written to the database in an optimized manner. This separation reduces unnecessary database operations and lowers system burden while maintaining data availability for analysis
Data Source
AI summary
The present application relates to a field of big data technology, in particular to a field of data storage technology. More specifically, the present disclosure relates to a method of importing data to a database, an electronic device, and a storage medium. A specific implementation solution is: acquiring incoming data from a data source according to a database config file; the incoming data is original data directly acquired from the data source; calculating and processing the incoming data according to the database config file to obtain computational data; the computational data is obtained by integrating and calculating the incoming data; writing the incoming data and the computational data into a database.