Flexible Data Ingestion System for Non-Standard Binary Formats
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database tables from various sources often cause runtime inefficiencies due to differences in binary formats and schemas, leading to inefficiencies in updating and querying ingested data, as standardization can sever relationships between original and rewritten tables.
Innovation Solution
A flexible data ingestion system that converts non-standard binary format database tables to a standard format without altering the original data, allowing for the generation of processed and materialized data chunks that maintain relationships and enable efficient querying while retaining the original data for updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If database tables are rewritten into a standard format, then querying efficiency is improved, but the relationship between the ingested database table and the rewritten database table is severed, causing inefficiencies in dealing with updates
Solution Approach 1:
The patent creates a copy of the ingested database table in standard format while preserving the relationship through metadata. The rewritten table is a copy that maintains traceability to the original through stored metadata, allowing efficient querying without losing the ability to track updates back to the source.
Solution Approach 2:
The patent introduces metadata as an intermediary that bridges the original ingested table and the rewritten standard format table. This metadata layer preserves the relationship information without requiring direct structural correspondence between the two tables, enabling both efficient querying and update tracking.
2Adaptability or versatility
If database tables from various sources are ingested with different binary formats and schemas, then data source compatibility is improved, but runtime inefficiencies occur due to format and schema differences
Solution Approach 1:
The patent segments the data handling process into distinct phases: ingestion phase where diverse formats are accepted and stored as-is, and querying phase where standardized views are provided. This segmentation allows compatibility with various data sources while maintaining runtime efficiency through pre-processing and standardization at appropriate stages.
Solution Approach 2:
The patent performs preliminary actions during the ingestion phase by storing metadata about each table's format and schema characteristics. This preliminary characterization enables efficient runtime querying without requiring real-time format conversion, as the standardization information is prepared in advance.
3Productivity
If the original ingested database table is altered to standard format, then querying efficiency is improved, but data integrity and the ability to handle updates efficiently are compromised
Solution Approach 1:
The patent creates a copy of the original data in standard format for efficient querying, while preserving the original ingested table unchanged. This copying approach maintains data integrity by keeping the source of truth intact while providing an optimized view for queries.
Solution Approach 2:
The patent uses metadata as an intermediary that links the original table and the standardized copy. This metadata layer preserves the relationship and enables update handling without requiring alteration of the original data, thus maintaining data integrity while achieving querying efficiency.
Data Source
AI summary
Systems and techniques are provided for flexible data ingestion. Data including a file including a database table may be received at a computing device. The file may be in a non-standard binary format. The data including the file may be stored unaltered as a source data chunk. A processed data chunk may be generated from the source data chunk by converting the file to a standard binary format and storing the file in the processed data chunk without altering the source data chunk. A materialized data chunk may be generated from the processed data chunk by performing, with a database server engine of the computing device, a database operation on the database table of the file of the processed data chunk and storing the file in the materialized data chunk without altering the processed data chunk. The database table of the file of the materialized data chunk may be made available for querying by the database server engine.