Database Cleanup System Using Attribute Grouping and Dynamic SQL
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data warehousing, identifying and correcting erroneous data is labor-intensive and memory-intensive due to the lack of notions of key separation, human-readable names, and attribute value evaluation in existing semantic net tools, which are not directly applicable to database contexts.
Innovation Solution
A system that extends the database metamodel with human-readable descriptions of entities, fields, and foreign key relationships, using 'bags' to group and filter database objects, and dynamically generates SQL statements for error correction, allowing for efficient identification and cleanup of erroneous data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If semantic net tools are used for data maintenance, then flexibility in handling incomplete data is improved, but applicability to database contexts deteriorates due to lack of key separation and human-readable names
Solution Approach 1:
The patent introduces a database-specific toolset that acts as an intermediary between the flexible semantic net concepts and the structured database environment. This toolset includes specialized components for handling database objects, maintaining referential integrity, and working with database-specific features like primary keys and foreign keys, thereby bridging the gap between semantic net flexibility and database structure requirements
Solution Approach 2:
The patent adapts the semantic net toolset by modifying its parameters and operations to be database-appropriate. Instead of treating all data uniformly as in semantic nets, the toolset now distinguishes between different data types, enforces database constraints, and utilizes database-specific operations while maintaining the high-level abstraction benefits
2Ease of operation
If semantic net tools perform mass operations in main memory, then ease of data manipulation is improved, but memory consumption increases causing failures on large data warehousing cases
Solution Approach 1:
The patent segments the data processing operation into multiple phases: first loading necessary data into memory for processing, then writing results back to the database. This segmentation allows the toolset to handle large datasets by processing them in manageable chunks rather than attempting to load entire datasets into memory at once, thereby reducing memory consumption while maintaining ease of operation
Solution Approach 2:
The patent creates a copy of the necessary data structures and metadata in memory for processing, rather than working directly with the entire database in memory. This copying approach allows the toolset to manipulate data representations in memory while the actual database remains on disk, significantly reducing memory requirements for large data warehousing cases
3Measurement precision
If manual SQL cleanup is performed, then precision in data correction is improved, but labor intensity and time consumption increase
Solution Approach 1:
The patent implements automated detection and correction of database errors through the toolset. The system automatically identifies data quality issues, determines appropriate corrections based on defined rules and constraints, and executes the corrections without requiring manual SQL intervention. This self-service approach maintains precision by adhering to database constraints and referential integrity while dramatically reducing the time and labor required compared to manual SQL cleanup
Data Source
AI summary
A system and method to efficiently perform database cleanup. Database objects are initially grouped based on selected attributes. The groups may be filtered to identify erroneous data within the group. Structure query language statements are then dynamically generated to directly correct the erroneous data within the database.