Self-Service Data Platform with In-Memory Query Acceleration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data analytics systems are designed exclusively for IT professionals, requiring time-consuming ETL processes and providing partial, stale data, and are inadequate for end-users seeking flexible access to diverse data sources.
Innovation Solution
A self-service data platform that enables users to discover, curate, and analyze data from various sources without ETL, using optimized data structures in memory to accelerate query execution and support diverse data types and sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If ETL pipelines are used to collect and store data in centralized data warehouses, then data can be centralized and structured, but query execution time increases and data becomes stale
Solution Approach 1:
The system performs preliminary actions by pre-computing and materializing query results in optimized data structures (e.g., inverted indexes, aggregated data models) before actual queries are executed. This allows fast query response without real-time ETL processing, resolving the contradiction between data structure stability and query speed.
Solution Approach 2:
The system creates optimized copies of data in specialized data structures (materialized views, caches, indexes) that replicate the necessary data transformations. These copies enable rapid query execution without repeatedly executing full ETL pipelines, maintaining both data structure integrity and query performance.
2Ease of operation
If BI tools connect to centralized data warehouses, then data access is simplified, but access to diverse distributed data sources is limited
Solution Approach 1:
The system introduces an intermediary layer (the self-service data platform) between BI tools and diverse distributed data sources. This intermediary abstracts the complexity of connecting to various data sources (cloud storage, NoSQL databases, Hadoop) and provides unified access through standardized interfaces, enabling both ease of operation and versatility.
Solution Approach 2:
The platform provides universal access to multiple data source types through a single unified interface. By implementing a universal query engine and data abstraction layer, the system allows BI tools to access diverse data sources (relational databases, NoSQL, cloud storage, Hadoop) without requiring separate connection mechanisms for each source type.
3Reliability
If data is stored in centralized data warehouses, then data consistency is improved, but data volume and complexity from distributed sources cannot be handled
Solution Approach 1:
The system segments data handling by maintaining distributed data sources as-is for large-scale storage while creating segmented optimized structures (materialized views, partitions, indexes) for query performance. This allows the system to handle massive data volumes from distributed sources while maintaining consistency through controlled synchronization points and materialization caches.
Data Source
AI summary
Disclosed embodiments include a method performed by server computer(s). The method includes receiving a query and defining a query plan based on the received query. The query plan refers to datasets contained in data sources. The method further includes determining that the received query can be accelerated based on an optimized data structure contained in a memory, where the optimized data structure is derived from a dataset referred to in the query plan. The method further includes modifying the query plan to include the optimized data structure, and executing the modified query plan to obtain query results that satisfy the received query by reading the optimized data structure in lieu of reading at least some data from the data sources.