Unified Query Interface for Big Data Storage Formats

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional relational databases struggle to handle large volumes of big data due to varying data storage formats, which require different interface approaches and query syntaxes, making it difficult for users to access and manage data across multiple formats efficiently.

Innovation Solution

A system that provides a unified, virtualized interface for accessing various big data storage formats by parsing requests, generating compatible queries, and managing access permissions, allowing users to interact with data using their preferred query language while ensuring compatibility with different storage formats.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If data is stored in multiple different data storage formats to service diverse application parameters, then the system can accommodate diverse application needs, but the difficulty of learning various interface protocols and adapting programs increases

Engineering Contradiction:

Improveability to service diverse application parametersVSAvoiddifficulty of learning interface protocols and query syntaxes

Core Design Contradiction:

Adaptability or versatilityVSEase of operation

Solution Approach 1:

The patent implements a universal query interface that can handle multiple data storage formats (Parquet, Avro, ORC, etc.) through a single standardized API. The system provides format-agnostic read/write operations, catalog-based data location, and unified query syntax that works across different storage formats, eliminating the need for users to learn multiple interface protocols while maintaining support for diverse formats.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent introduces an intermediary layer (the file system interface and catalog system) between the user and the diverse data storage formats. This intermediary translates standardized user requests into format-specific operations, hiding the complexity of different query syntaxes and interface protocols from users while maintaining compatibility with various storage formats underneath.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Quantity of substance

If traditional relational databases are used to store big data, then the data can be organized in structured tables, but the databases cannot sufficiently handle the size of big data tables

Engineering Contradiction:

Improvevolume of dataVSAvoidcapability to handle table size

Core Design Contradiction:

Quantity of substanceVSReliability

Solution Approach 1:

The patent segments big data into columnar storage formats (Parquet, Avro, ORC) that are optimized for large-scale data processing. Instead of using traditional row-based relational database tables, the system divides data into columnar files organized in distributed file systems, enabling efficient handling of billions of records and hundreds of thousands of columns while maintaining data organization through catalogs and metadata.

Inventive Principle:

Principle #1Segmentation

3Adaptability or versatility

If data is stored in various data storage formats in various locations, then diverse application parameters can be serviced, but the complexity of managing diverse data sources increases

Engineering Contradiction:

Improveability to service diverse application parametersVSAvoidcomplexity of managing diverse data sources

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent introduces a catalog system as an intermediary that manages diverse data sources. The catalog stores metadata about data location, format, and structure, allowing the system to manage multiple storage formats and locations through a unified interface. This intermediary layer abstracts the complexity of diverse data sources while maintaining the ability to service diverse application parameters.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent implements a universal file system interface that can read and write multiple data storage formats (Parquet, Avro, ORC, JSON, etc.) through a single standardized API. This universal interface handles format detection, data location through catalogs, and standardized operations, reducing the complexity of managing diverse data sources while maintaining support for multiple formats and locations.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11620400B2Querying in big data storage formats

Publication Date: 2023.04.04 AMERICAN EXPRESS TRAVEL RELATED SERVICES CO INC

AI summary

A system may receive a request for access to a first variable. The request may include a requested action and a variable identifier for the first variable. The request may also have a syntax that is incompatible with first data storage format. The system may parse the request to capture the variable identifier from the request. The system may also look up a location of the first variable in a catalog using the variable identifier. The location of the first variable may include the first data storage format. The system may generate a query to access the first variable. The syntax of the query may be compatible with the first data storage format. The system may then submit the query to access the first data storage format. The query may be configured to complete the requested action.