Big Data Feature Extraction via Query Family Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods are impractical for extracting and generating large sets of features from vast Big Data systems, particularly in domains with temporal relations and complex data, due to the impracticality of running queries for each feature.
Innovation Solution
A method and software that provide a plurality of queries grouped into families, including simple and complex query families, to retrieve and process data from Big Data systems, synthesizing features that characterize answers by performing a single pass through the data and using results from simple queries as filters for complex queries, thereby reducing computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a straightforward approach of running a query per feature is used to extract features from Big Data systems, then the completeness and richness of the feature set is improved, but the computational complexity and time consumption becomes impractical
Solution Approach 1:
The patent segments the feature extraction process by dividing features into different families (e.g., demographic features, behavioral features, consumption features) and processing each family through dedicated query templates. This allows parallel processing of multiple feature families simultaneously, transforming the impractical single-query-per-feature approach into an efficient family-based batch processing system that maintains feature completeness while dramatically improving extraction speed
Solution Approach 2:
The patent performs preliminary actions by pre-defining query templates and feature families before actual data extraction. The system pre-processes the feature specification, organizing queries into families and preparing the extraction framework in advance. This preliminary structuring enables the system to handle large-scale feature extraction efficiently by avoiding ad-hoc query formulation during the actual extraction phase
2Reliability
If a rich universe of features is created by extracting and generating many features, then the quality and effectiveness of machine-learning processes is improved, but the device complexity and computational resources required increase
Solution Approach 1:
The patent segments features into hierarchical families (e.g., subscriber features, call features, messaging features, consumption features) with each family processed by specialized query templates. This segmentation allows the system to manage complexity through modular organization while still generating a comprehensive feature universe, as each segment can be independently optimized and processed
Solution Approach 2:
The patent creates universal query templates that can generate multiple features across different families using the same underlying query structure. For example, a single template pattern can generate various temporal aggregation features (daily, weekly, monthly statistics) across different feature types, reducing system complexity through reusability while maintaining feature richness
3Quantity of substance
If features are extracted and generated in large quantities without prior information on which features are important, then the completeness of the feature set is improved, but the time and computational resources required become impractical
Solution Approach 1:
The patent performs preliminary classification of features into families based on their potential importance and relevance to common machine-learning tasks. By pre-organizing features into logical families (demographic, behavioral, consumption, etc.) and assigning appropriate query templates in advance, the system can efficiently extract comprehensive feature sets without wasting time on irrelevant features, as the framework is already structured to focus on meaningful feature categories
Data Source
AI summary
A method is provided for enabling a software user to obtain answers based on information retrieved from Big Data systems to complex questions, which comprises the steps of: providing a plurality of queries associated with different query families, the different query families comprise at least one family of simple queries and at least one query family of complex query family, and each of the families is associated with the following characterizing elements: information sources, entity type for which one or more features would be synthesized, attributes to be used as filters and attributes for calculating the new features; retrieving data from the respective information sources; processing the retrieved data to enable evaluating results of the queries associated with the at least one simple query families; calculating solutions to all queries and synthesizing features characterizing the answers, based on the calculated solutions.