Data Sessionization Pipeline for Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for sessionizing data, which involve combining session data feeds and ETL data feeds, are resource-intensive and time-consuming, as they require processing large amounts of data from previous intervals to compute metrics for future intervals, often sending unnecessary data and consuming significant network bandwidth and computing resources.
Innovation Solution
A system that extracts and combines source data and session data to create a single data pipeline, which is then sessionized into finished and unfinished events intervals, allowing for the publication of a single data feed that includes only necessary data, reducing resource usage and improving processing speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If session data feed and ETL data feed are combined to sessionize data, then complete user interaction data is available for metric computation, but network bandwidth and computational resources are excessively consumed
Solution Approach 1:
The patent segments the data processing workflow into distinct components: session data feed processing, ETL data feed processing, and metric computation. By dividing the data pipeline into intervals (current interval, future interval, last interval) and processing only relevant segments, the system avoids transmitting and processing all data, thus reducing network bandwidth and computational resource consumption while maintaining data completeness where necessary.
Solution Approach 2:
The patent extracts only the necessary subset of data from the complete session data feed and ETL data feed. Specifically, it extracts events from the current interval that are needed for future interval metric computation, and events from the last interval that are needed for current interval metrics. This extraction principle eliminates unnecessary data transmission and processing, directly reducing network bandwidth and computational resource usage.
2Measurement precision
If all data from current interval is sent to process future interval metrics, then complete information is available for computation, but processing time and resource usage increase significantly
Solution Approach 1:
The patent performs preliminary actions by pre-processing and organizing data into session data feeds and ETL data feeds with proper time interval annotations before metric computation. This preliminary organization allows the system to quickly identify and extract only the necessary data subsets for each metric computation task, avoiding the need to process all raw data and significantly reducing processing time while maintaining computation accuracy.
Solution Approach 2:
The patent applies partial action by computing metrics using only the necessary subset of data rather than all available data. For example, when computing metrics for the future interval, it uses only events from the current interval that are relevant, not all events from the current interval. This partial processing approach reduces processing time and resource usage while maintaining sufficient measurement precision for accurate metric computation.
3Reliability
If session data and source data are processed separately, then data integrity is maintained, but system complexity and processing overhead increase
Solution Approach 1:
The patent merges the session data feed processing and ETL data feed processing into a unified data pipeline architecture. Both data feeds are ingested and processed together through a single pipeline that handles sessionization, interval segmentation, and metric computation. This merging maintains data integrity by preserving the separate identities and characteristics of each data feed while simplifying the overall system architecture and reducing processing overhead compared to completely separate processing systems.
Data Source
AI summary
Users may interact with a multitude of content on content sources. Content sources may desire to identify sessions that the user created. As provided herein, a user interacting with a content source may be identified. Source data and session data regarding the user interaction may be extracted. A data pipeline may be generated based upon the source data and the session data. The data pipeline may be sessionized into a single data feed based upon a finished current events interval (e.g., sessions that are activated and terminated during a first interval), a finished last events interval (e.g., sessions that are activated during the first interval and terminated during a second interval), a second finished current events interval (e.g., sessions that are activated and terminated during the second interval), and/or a second finished last events interval (e.g., sessions that are activated during the second interval and terminated during a third interval).