File System to Tabular Database Mapping for Automated Data Import
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The cumbersome process of bulk data upload into tabular databases requires frequent file specification, often leading to forgotten operations due to constant data streams and frequent updates, making it inefficient for users.
Innovation Solution
A storage system that creates a mapping between a file system location and a database table, allowing users to automatically upload data files to a specific database table without additional queries, using filesystem and database interfaces to manage file migration and tracking, with rules for file types and deletion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual bulk upload commands are used to load data into database tables, then data can be imported into the tabular database, but the process becomes cumbersome and users frequently forget to perform uploads due to constant data streams
Solution Approach 1:
The system pre-establishes a mapping between file system locations and database tables before data upload occurs. This preliminary configuration enables automatic recognition and import of data files without requiring users to manually specify target tables each time, eliminating forgotten uploads while maintaining operational simplicity
Solution Approach 2:
The system performs self-service by automatically monitoring the file system for new files, identifying their types, and importing them into the corresponding database tables based on pre-defined mappings. This automation removes the need for manual bulk upload commands, ensuring reliable data import without user intervention
2Reliability
If automatic monitoring of file system is implemented, then data upload reliability improves, but system complexity increases
Solution Approach 1:
The system employs a universal file type classification mechanism that handles multiple file formats (CSV, JSON, XML) through a single automated import pipeline. This multi-functional approach achieves reliable automatic data import while avoiding the need for separate complex systems for each file type, thus limiting complexity increase
Data Source
AI summary
A method for importing data to a tabular database (TD), the method includes: determining that data was written, using a file system command, to a file system location; wherein the file system location is associated with at least a part of the TD; and automatically importing the data to the at least part of the TD.