Object Table Querying for Unstructured Data Warehouses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analytics programs are rigid and cannot handle unstructured data without manual conversion, incurring high costs and difficulty in maintaining lineage information.
Innovation Solution
Implementing an object table system that organizes unstructured data into a tabular format, enabling analytics programs to process unstructured data by generating structured data tables using machine learning models and data scrapers, with row access policies for security.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If manual conversion of unstructured data into structured data is performed, then analytics programs can process the data, but implementation costs increase and data lineage maintenance becomes difficult
Solution Approach 1:
The patent introduces an object table as an intermediary structure between unstructured data storage and analytics processing. The object table stores metadata about unstructured data files in a structured format, enabling analytics programs to query and process unstructured data without manual conversion. This mediator layer resolves the contradiction by providing adaptability to handle unstructured data while maintaining implementation simplicity through automated metadata management.
2Productivity
If analytics programs are designed to handle only structured data, then processing efficiency is maintained, but adaptability to handle unstructured data is lost
Solution Approach 1:
The patent makes the analytics program universal by enabling it to handle both structured and unstructured data through the object table mechanism. The object table allows the same analytics program to query unstructured data files using standard SQL-like syntax, providing multi-functionality without sacrificing processing efficiency. The system maintains productivity by automating the data retrieval and presentation processes.
3Device complexity
If unstructured data is stored without structured metadata, then storage simplicity is maintained, but data organization and query capability are reduced
Solution Approach 1:
The patent segments unstructured data into two components: the actual unstructured data files stored in data lakes or repositories, and structured metadata stored in the object table. This segmentation allows the system to maintain storage simplicity for the unstructured data while providing structured query capability through the metadata layer. The object table columns organize data by file properties, enabling easy querying without complicating the underlying data storage.
Data Source
AI summary
A method for unstructured data analytics in data warehouses includes receiving an unstructured data query from a user, the unstructured data query requesting the data processing hardware determine one or more unstructured data files stored at a data repository that match query parameters. The method includes determining, using an object table, a set of unstructured data files stored at the data repository that matches the query parameters. The object table includes a plurality of rows, each row of the plurality of rows associated with a respective unstructured data file stored at the data repository, and a plurality of columns, each column of the plurality of columns comprising metadata associated with the respective unstructured data file of each row of the plurality of rows. The method includes returning, to the user, a structured data table including the determined set of unstructured data files.