Webgraph Compression of Data Records for Faster Impression Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for storing and accessing large amounts of impression data, such as those collected by promotion and marketing services, are inefficient and ineffective, leading to challenges in storage capacity and data access speed.
Innovation Solution
The implementation of webgraph-based techniques for compressing data records, which represent impression data, user location information, and application logs, by generating a directed link graph and using properties like locality and similarity to create a compressed list of records, thereby improving storage efficiency and data access speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If current storage methods are used for impression data, then data can be stored, but storage capacity is limited and data access speed is slow
Solution Approach 1:
The patent segments the impression data into multiple components (e.g., consumer identifier, impression identifier, timestamp, position information) and organizes them into a structured format. This segmentation allows for efficient compression by identifying and exploiting patterns within each component type, thereby improving both storage capacity and access speed.
Solution Approach 2:
The patent transforms the traditional flat storage structure into a multi-dimensional representation by organizing data along multiple axes (consumer dimension, impression dimension, temporal dimension, positional dimension). This dimensional transformation enables more efficient compression algorithms to be applied and allows for faster data retrieval by querying along specific dimensions without reading entire records.
2Quantity of substance
If data records are compressed using webgraph-based techniques, then storage efficiency improves, but data access requires transformation from graph format
Solution Approach 1:
The patent performs preliminary organization of data into a webgraph structure during the data ingestion phase, rather than transforming data at query time. The impression data is pre-processed to create the directed link graph with consumers as source nodes and impressions as target nodes, establishing the compressed representation in advance. This preliminary action eliminates the need for complex transformations during data access operations.
Solution Approach 2:
The patent introduces an intermediary layer that maintains the webgraph structure as a compressed representation of the original impression data. This intermediary graph structure acts as a mediator between the raw data and the query processing system, enabling efficient storage while providing optimized access paths for common query patterns through the graph's inherent structure.
3Loss of time
If multiple I/O operations are performed for data retrieval, then complete data can be accessed, but access time increases
Solution Approach 1:
The patent merges multiple data components that would traditionally require separate I/O operations into a single compressed webgraph structure. By combining consumer data, impression data, timestamps, and position information into an integrated graph representation, the system can retrieve related data in a single I/O operation rather than performing multiple sequential reads, thereby reducing access time and improving I/O efficiency.
Solution Approach 2:
The patent implements a nested structure where impression data is nested within consumer contexts in the webgraph. Each consumer node contains or references multiple impression nodes, allowing the system to retrieve all impressions for a consumer in a single operation. This nesting eliminates the need for separate I/O operations to fetch consumer information and their associated impressions, as they are hierarchically organized and stored together.
Data Source
AI summary
In general, embodiments of the present invention provide systems, methods and computer readable media for data record compression using graph-based techniques.