Automatic Feature Computation from Data Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analytics applications face challenges in efficiently computing and updating aggregate features from data streams, particularly in real-time, due to changes in data schema and business intelligence needs, which require significant development and resource allocation, and can be affected by server failures or workload imbalances.
Innovation Solution
A system that automatically computes aggregate features from a data stream by storing multiple copies of data across distinct hardware storage devices, allowing for resilient data ingestion and rollup operations, with dynamic server scaling and configuration changes to manage workload and schema updates without locks, ensuring timely computation and storage within predefined thresholds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple copies of data are stored across distinct hardware storage devices, then system reliability is improved, but device complexity increases
Solution Approach 1:
The patent implements data redundancy by storing multiple copies of data across distinct hardware storage devices. This copying approach ensures that if one storage device fails, the system can still access data from other copies, thereby improving reliability while managing complexity through standardized copy management procedures
2Productivity
If rollup operations are performed to compute aggregate features in real-time, then productivity is improved, but use of energy increases
Solution Approach 1:
The system performs rollup operations proactively to pre-compute aggregate features before they are requested. By maintaining a rolling window of data and continuously updating aggregates, the system avoids expensive real-time computations when queries are made, thereby improving productivity while managing energy consumption through selective pre-computation
3Adaptability or versatility
If the schema of transactional data changes, then adaptability is improved, but device complexity increases
Solution Approach 1:
The patent implements a dynamic schema management system that allows transactional data schemas to evolve over time. The system can add, remove, or modify data fields without requiring complete system reconfiguration, enabling adaptability while managing complexity through automated schema migration and validation mechanisms
4Measurement precision
If retroactive feature computation is performed on historical data, then measurement precision is improved, but loss of time increases
Solution Approach 1:
The system continuously maintains rolling aggregate features as data is ingested, performing rollup operations incrementally rather than computing everything retroactively when needed. This preliminary computation approach ensures accurate feature values are already available when historical analysis is required, improving measurement precision while minimizing time loss
Data Source
AI summary
Implementations described herein relate to methods, systems, and computer-readable media to automatically compute features from a data stream. In some implementations, a method includes receiving a data stream comprising a plurality of data fields. The method further includes storing at least three copies of data in the data stream in a database, wherein each copy is stored on a corresponding hardware storage device. The method further includes performing, one or more rollup operations on a first copy of the data to compute one or more aggregate features and storing the one or more aggregate features on each of the hardware storage devices. The method further includes receiving a request for a particular aggregate feature and providing the particular aggregate feature in response to the request.