Unified Data Pre-Processing for Search Indexes and ML Pipelines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database management systems (DBMS) face inefficiencies due to separate user interfaces for search and machine learning sub-systems, leading to redundant text pre-processing, errors, and reduced throughput, especially in query quota-based systems.
Innovation Solution
Implement a common specification in the DBMS that can be parsed for both search indexing and machine learning pre-processing, allowing unified text pre-processing across both systems, reducing errors and improving efficiency by ensuring consistent data treatment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If separate user interfaces are developed for search and machine learning sub-systems, then each sub-system can be optimized for its specific requirements, but redundancy and errors occur due to separate text pre-processing
Solution Approach 1:
The patent merges the separate user interfaces for search and machine learning sub-systems into a unified interface that accepts a common specification format. This allows both sub-systems to receive pre-processed text through the same interface, eliminating redundant pre-processing operations and reducing errors while maintaining the ability to optimize for specific requirements through specification parameters.
Solution Approach 2:
The patent creates a universal specification format that can be used by both search and machine learning sub-systems. This multi-functional interface accepts requests with specifications containing text, parameters, and options that can be interpreted by different sub-systems, allowing a single interface to serve multiple purposes without sacrificing sub-system-specific optimization.
2Adaptability or versatility
If separate text pre-processing is performed for search and machine learning, then each sub-system can apply its own requirements, but processor usage and data traffic increase due to duplication
Solution Approach 1:
The patent implements preliminary text pre-processing in the common interface before distributing requests to specific sub-systems. By performing initial text cleaning, tokenization, and parameter extraction once in the unified interface, the system avoids duplicate pre-processing operations while still allowing each sub-system to apply its specific requirements through specification parameters, thereby reducing processor usage.
Solution Approach 2:
The patent introduces a common specification format as an intermediary between the user interface and the search/machine learning sub-systems. This intermediary layer performs initial text pre-processing and then distributes the pre-processed text with appropriate parameters to the relevant sub-systems, eliminating redundant processing while maintaining adaptability to specific sub-system requirements.
3Productivity
If a common specification is implemented for both search and machine learning, then efficiency improves through unified pre-processing, but customization depth for search indexes may be reduced
Solution Approach 1:
The patent implements a dynamic specification structure that can be configured at multiple levels. The common specification format includes optional parameters that allow users to specify different pre-processing depths and types based on the intended use. This dynamic configuration enables the same interface to provide both simple unified pre-processing for efficiency and detailed customized pre-processing when needed, resolving the contradiction between productivity and customization.
Data Source
AI summary
Methods, systems, and apparatus, including computer-readable storage media relating to a database management system (DBMS) configured to perform data searching and machine learning (ML) pre-processing using a common user specification. Two common functions associated with large databases are searching for data in the database and using data stored in databases for data-intensive processing, such as training or executing a machine learning model. While a single DBMS may implement separate sub-systems for searching and machine learning model processing that rely on similar operations, e.g., text processing, the respective interfaces for each sub-system are different and have different requirements for properly formed user input. A specification, when parsed by either a search sub-system or ML pre-processing system, can allow for correctly pre-processing data in accordance with the specification, without the user having to provide separate specifications for either sub-system.