Protocol Buffer Trie Packing for Columnar Database Payload Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Columnar databases face challenges with large update and insert payloads that overwhelm network resources, necessitating a reduction in payload size for efficient transmission and improved performance during data updates.
Innovation Solution
The protocol buffer trie packing and streaming update method involves creating a trie data structure to deduplicate coordinate information, grouping frequent terms, and serializing the trie into a compact protobuf format for efficient transmission and processing, allowing multiple field updates simultaneously.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional data update methods are used in columnar databases, then data can be updated and inserted, but the payload size becomes large and overwhelms network resources
Solution Approach 1:
The patent segments the data update payload into multiple components: coordinate information, field identifiers, and value data. By organizing these segments into a hierarchical structure with coordinate prefixes and suffixes, the system eliminates redundant coordinate repetitions across multiple field updates, significantly reducing overall payload size while maintaining data integrity
Solution Approach 2:
The patent implements a nested structure where coordinate information is packed hierarchically: common prefix coordinates are shared across multiple fields, intermediate coordinates are grouped by field, and leaf coordinates specify individual data points. This nested organization allows efficient compression by nesting shared coordinate information within field-specific update groups
2Ease of operation
If coordinate information is repeated for each field update, then field updates can be specified, but the payload size increases significantly
Solution Approach 1:
The patent merges repeated coordinate information into a single shared prefix that is referenced by multiple field updates. Instead of specifying full coordinates for each field update, the system combines common coordinate portions into a prefix and only transmits the differing suffix portions, dramatically reducing payload size while preserving complete update specifications
Solution Approach 2:
The patent creates a coordinate prefix structure that acts as a template or copyable reference for multiple field updates. Rather than transmitting identical coordinate information repeatedly, the system transmits the prefix once and references it across multiple field update operations, eliminating redundancy while maintaining operational clarity
3Productivity
If update payloads are transmitted over the network, then data can be updated, but network resources are overwhelmed by large payload sizes
Solution Approach 1:
The patent changes the structural parameters of the update payload by introducing hierarchical coordinate organization with prefixes and suffixes. This parameter change transforms the payload from a flat, repetitive structure to a compressed hierarchical structure, reducing the amount of data that must be transmitted over the network while preserving all necessary update information
Data Source
AI summary
Systems and techniques for protocol buffer trie packing and streaming updates are described herein. Data entities are identified in a received update query. An index value is assigned to each unique data entity to generate an index of unique data entities. A frequency of occurrence value is determined for each unique data entity. The index of unique data entities is sorted by data entity type in descending order by the frequency of occurrence value of each unique data entity. A trie is generated with a root node assigned as a first unique data entity from the index of unique data entities that has a frequency of occurrence value with the largest numerical value. An update payload is created for the trie by conversion of the trie into a protocol buffer format through serialization of the trie into a binary string.