Schema Inference for Data Input Files
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems require users to manually specify the schema for data input files, which limits usability and prevents handling of diverse and older file formats.
Innovation Solution
A computer-implemented system that infers a schema for a data input file by analyzing the file to identify row delimiters, column delimiters, and data types, thereby generating a candidate schema automatically.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If users manually specify the schema for data input files, then the system can correctly process and store the data, but the usability is limited and requires user expertise in file structure
Solution Approach 1:
The system performs self-service by automatically inferring the schema from the data input file itself. The server computer analyzes the file to identify delimiters, data types, and structure without requiring user input or expertise, thereby resolving the contradiction between reliable data processing and ease of operation
Solution Approach 2:
The system performs preliminary action by pre-analyzing the data input file to automatically determine its schema before processing. This preliminary schema inference eliminates the need for users to manually specify schemas, improving usability while maintaining processing correctness
2Device complexity
If the system requires uniform schema for all uploaded files, then schema processing is simplified, but the system cannot handle diverse file formats including older files
Solution Approach 1:
The system applies dynamics by making the schema adaptive rather than static. Instead of requiring a fixed uniform schema, the system dynamically infers the appropriate schema for each file based on its specific structure, enabling handling of diverse file formats while maintaining manageable processing complexity
Solution Approach 2:
The system applies local quality by allowing each data input file to have its own specific schema characteristics inferred from its local structure. This enables the system to handle diverse file formats with different structures without requiring global uniformity, thereby improving file format compatibility
3Ease of operation
If automatic schema inference is implemented, then system usability is improved and diverse file formats can be handled, but the system must analyze and identify multiple schema components
Solution Approach 1:
The system applies segmentation by breaking down the schema inference process into distinct analytical components: identifying delimiters, determining data types, detecting structure, and inferring encoding. This segmentation manages the complexity of automatic schema inference while enabling comprehensive analysis for improved usability
Data Source
AI summary
A method comprises selecting a sample excerpt from a data input file; in response to the determining that a first row in the sample excerpt does not contain a delimited value and a second row does contain a delimited value, determining that the first row consists of header data; identifying one or more jagged rows based on row delimiters that were erroneously placed; causing displaying text that led to creation of a jagged row; receiving an addition or removal of a specific row delimiter to the text; updating the sample excerpt based on the addition or the removal; analyzing the sample excerpt to determine a row delimiter for the data input file; identifying a plurality of rows that is not included in the header data; identifying a plurality of candidate column delimiters and generating a candidate schema for the data input file.