Parsing Unstructured Data Tokens into Structured Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods fail to automatically assign headers and unambiguously assign data values to unstructured data, hindering the formation of structured data tables essential for scientific and industrial processes.
Innovation Solution
A system and method that apply a configuration construct to tokens in a data construct, labeling subsets as header and value tokens based on position and delimiters, transforming the data into a structured table with each token serving as a header or data value in columns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If delimiters are used to indicate data field boundaries in unstructured data, then data parsing can be performed, but the ability to automatically assign headers and unambiguously assign data values to headers is absent
Solution Approach 1:
The patent introduces an intermediary layer (parsing system with configuration constructs) between the raw unstructured data and the final structured data table. This intermediary automatically recognizes patterns, assigns headers based on positional relationships and delimiters, and maps data values to appropriate headers without human intervention, thereby achieving automatic header assignment while preserving data structure information.
Solution Approach 2:
The system changes the parameter of data representation by transforming unstructured data with delimiters into structured data tables with defined headers and values. By applying configuration constructs that specify parsing rules, the system automatically identifies header positions and data value positions based on parameter changes in data format and structure.
2Productivity
If manual parsing methods are used to create structured data tables, then data organization can be achieved, but the process is time-consuming and labor-intensive
Solution Approach 1:
The parsing system performs self-service by automatically analyzing unstructured data, identifying delimiters, determining header positions, and creating structured data tables without requiring manual intervention. The system uses configuration constructs to guide the automatic parsing process, enabling high-speed data transformation while eliminating time-consuming manual operations.
Solution Approach 2:
The patent replaces manual mechanical parsing operations with an automated computational system. Instead of manually examining and organizing data fields, the system uses algorithmic processing to automatically parse unstructured data based on configuration constructs, significantly increasing productivity while reducing the time required for data processing.
3Ease of manufacture
If unstructured data is used directly for analysis, then data storage is simplified, but data organization and analysis capability are hindered
Solution Approach 1:
The patent applies segmentation by dividing unstructured data into distinct structured components (headers and data values) organized in columns. This segmentation transforms the data from an indistinguishable mass into clearly defined, analyzable units while maintaining storage efficiency. The configuration constructs guide this segmentation process to create data tables suitable for analysis.
Solution Approach 2:
The system transitions data from a one-dimensional unstructured format to a two-dimensional structured table format with rows and columns. This dimensional change organizes data into a grid structure where headers define columns and values populate rows, dramatically improving data analysis capability while preserving the underlying information.
Data Source
AI summary
Systems and methods for forming a data table are provided. A data construct comprising a plurality of tokens is obtained. Application of a configuration construct to the tokens labels a first subset of the tokens as header tokens and a second subset of the tokens as value tokens based on the positions of tokens in the first and second subsets in the data construct. Application of the configuration construct associates specific tokens in the first subset to tokens in the second subset based on the relative or absolute position of tokens and/or by delimiters between tokens in the first and second subsets in the data construct. The associating transforms the data construct into the data table in which each token in the first subset is a header and each token in the second subset is a data value in a single column in the data table.