Backend Data Aggregation System Using Dynamic ORM and Columnar Store
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional technologies for managing very large enterprise customer experience data face challenges in balancing query speed and flexibility, leading to inefficient data processing and long load times, especially in handling large volumes of records and frequent data structure changes.
Innovation Solution
A backend data aggregation system utilizing a Kafka data backbone, combining a columnar data store and a dynamic Object-Relational Mapper (ORM) with a dynamic Extract Transform Load (ETL) process, enables efficient data transformation and storage, allowing for scalable and flexible data querying and reporting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data cube technology (SQL Server, Star Schema) is used, then query speed is improved, but flexibility and scale are insufficient for large enterprise customer experience programs
Solution Approach 1:
The system segments data storage and processing into multiple specialized components: a columnar data store for optimized query performance, a NoSQL data store for flexible schema handling, and a stream processing layer for real-time data transformation. This segmentation allows each component to excel at its specific function while working together to provide both speed and flexibility.
Solution Approach 2:
The patent creates a composite data architecture that combines multiple database technologies (columnar storage, NoSQL, stream processing) into a unified system. This composite approach leverages the strengths of each technology type to achieve both fast query performance and high adaptability for enterprise customer experience programs.
2Adaptability or versatility
If big data technology (Hadoop, NoSQL, Elastic Search) is used, then flexibility to process thousands of data sources is improved, but query speed for fast load times deteriorates
Solution Approach 1:
The stream processing layer acts as an intermediary between the flexible NoSQL data store and the query interface. It pre-processes and transforms data in real-time, converting unstructured or semi-structured data into optimized formats that can be quickly queried, thus bridging the gap between flexibility and speed.
Solution Approach 2:
The system performs preliminary data transformation and structuring through stream processing before data is queried. By pre-processing data as it enters the system and maintaining it in optimized formats, the architecture eliminates the need for slow post-query processing while preserving schema flexibility.
3Quantity of substance
If conventional technology platforms are used for large data imports, then data processing capacity is limited, but system complexity and processing time increase
Solution Approach 1:
The stream processing architecture enables continuous data ingestion and transformation without batch processing interruptions. Data flows continuously through the system, being transformed and stored in real-time, which eliminates the stop-start nature of conventional batch processing and significantly reduces overall processing time for large data volumes.
Solution Approach 2:
The system employs dynamic, real-time data transformation through stream processing rather than static batch processing. The architecture adapts to incoming data flows dynamically, processing and transforming data as it arrives, which enables handling of large data volumes with reduced latency and processing time.
4Reliability
If data is reloaded or extensive changes are made in present CX technology, then data accuracy is maintained, but reprocessing every data point causes weekend job delays
Solution Approach 1:
The stream processing layer extracts and processes only the specific data that has changed or needs updating, rather than reprocessing entire data sets. This selective processing approach maintains data accuracy by applying transformations only where needed, dramatically reducing reprocessing time while preserving data integrity.
5Ease of manufacture
If survey files are broken up into smaller files for sending, then data transmission is manageable, but send times extend to hours
Solution Approach 1:
The stream processing architecture enables continuous, real-time data transmission without breaking files into smaller batches. Data flows continuously through the system in an optimized stream, eliminating the need to fragment large survey files and reducing transmission time from hours to near-real-time operations.
Data Source
AI summary
A backend data aggregation and customer experience (CX) systems and methods for processing very-large enterprise CX data. A columnar data store has a transform processor coupled thereto via a data communication network. The transform processor is configured to receive customer experience data from a plurality of data sources and to dynamically transform the received data into a columnar data schema format for storage in the columnar data store. A dynamic Object-Relational Mapper (ORM) is configured to dynamically map the transformed data to structured data based on metadata associated therewith and to store the structured data in the columnar data store.