Interaction Data Stream Metric Determination Using Sharded Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional processing systems face performance degradation when handling large volumes of data streams from sources like SaaS platforms, leading to inefficiencies in generating real-time metrics due to the need for multiple queries across entire datastores.
Innovation Solution
A system that uses a non-relational database for sharding interaction data based on interaction categories and time intervals, with a relational database storing intermediate metrics to calculate cumulative metrics efficiently, allowing for real-time updates and reduced processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional processing systems store and process large-volume data streams in a single datastore, then data integrity is maintained, but system performance degrades and query time increases
Solution Approach 1:
The patent divides the single datastore into multiple sharded datastores (first sharded datastore, second sharded datastore, etc.), each storing a portion of the interaction data. This segmentation allows parallel processing and reduces the time required to query and generate metrics, directly addressing the performance degradation issue while maintaining data integrity across distributed storage.
Solution Approach 2:
The patent introduces an intermediary processing system that aggregates data from multiple sharded datastores and computes metrics. This intermediary layer decouples the data storage from the metric computation, enabling efficient real-time metric generation without requiring direct access to the entire distributed datastore, thus reducing query time while maintaining accuracy.
2Measurement precision
If multiple queries are made over an entire datastore to generate real-time reports, then comprehensive metrics are obtained, but resource overload occurs and processing efficiency decreases
Solution Approach 1:
By segmenting the datastore into multiple sharded components and assigning specific interaction categories to specific shards, the system enables targeted queries on individual shards rather than scanning the entire datastore. This maintains metric accuracy for each category while dramatically improving processing throughput by reducing the data volume that must be queried.
Solution Approach 2:
Each sharded datastore is optimized for storing specific types of interaction data (e.g., first sharded datastore for first interaction category, second sharded datastore for second interaction category). This local optimization allows each shard to be queried independently and efficiently, maintaining comprehensive metric accuracy while avoiding the resource overload of querying a monolithic datastore.
3Adaptability or versatility
If a single database stores all interaction data, then data consistency is maintained, but system complexity increases and scalability is limited
Solution Approach 1:
The patent implements a segmented database architecture where interaction data is distributed across multiple sharded datastores based on interaction categories. This segmentation enables independent scaling of each shard and allows the system to adapt to growing data volumes without requiring a complete architectural overhaul, thus improving scalability while managing complexity through modular design.
Solution Approach 2:
The patent creates a universal metric processing system that can handle multiple types of interaction data across different sharded datastores through a common aggregation and computation framework. This multi-functional approach allows the same processing logic to operate on diverse data types, enhancing system adaptability while maintaining manageable complexity through standardized interfaces.
Data Source
AI summary
Techniques are disclosed for generating metrics for a stream of interaction data. In some embodiments, a computing system receives a stream of interaction data that is based on user interactions with one or more computing systems, where the stream of interaction data includes information for multiple interaction characteristics including an interaction category. In response to receiving additional interaction information in the stream of interaction data, the computing system may generate cumulative metric values for one or more interaction categories. Generating cumulative metric values may include: sharding the additional interaction information based on multiple characteristics including interaction category, storing the sharded information in a non-relational database, determining respective metric values for shards modified by the storing, storing the respective metric values and shard information for each of the modified shards in a relational database, querying the relational database using one or more interaction categories corresponding to the modified shards, and determining the cumulative metric values based on the results of the querying.