Column Dictionary Data Transform Intersection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The processing cost of data transforms in enterprise database systems increases proportionally with the number of input records, as existing methods process records sequentially, leading to inefficiencies in data standardization and transformation.
Innovation Solution
The implementation of a data transformation process that utilizes column dictionaries and hashmaps to efficiently identify and process records by intersecting values across columns, reducing the need for independent transformations and leveraging previously generated outputs to accelerate processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If records are processed sequentially using traditional transform methods, then data standardization and transformation can be completed, but the processing cost increases proportionally with the number of input records
Solution Approach 1:
The patent creates column dictionaries and hashmaps in advance before processing records. These data structures pre-compute and store the relationships between column values and record identifiers, allowing the transform operation to quickly lookup and identify matching records without performing expensive computations during the actual transformation process. This preliminary preparation significantly reduces the time required per record during batch processing.
Solution Approach 2:
The patent creates simplified copy representations of the data through hashmaps that map column values to record identifiers. Instead of processing the full complex record structures sequentially, the system uses these lightweight hashmap copies to quickly identify matching records, reducing the computational overhead while preserving the essential matching logic needed for the transformation.
2Productivity
If traditional sequential processing is used, then all input records can be transformed, but memory usage increases with the need to process each record independently
Solution Approach 1:
The patent segments the data processing into two distinct phases: (1) building column dictionaries and hashmaps from the input data, and (2) using these structures to efficiently identify and transform matching records. This segmentation allows the system to reuse the same pre-computed structures across multiple records without reloading or recomputing, reducing overall memory usage while maintaining high transformation throughput.
Solution Approach 2:
The column dictionaries and hashmaps serve multiple functions: they store column value to record identifier mappings, enable quick lookup of matching records, and support the transform operation across all input records. This multi-functionality eliminates the need for separate data structures for each processing step, reducing total memory consumption while maintaining processing efficiency.
Data Source
AI summary
A system includes acquisition of values of a plurality of columns of a first record of a set of records, generation of a first output based on the values, association of the first output with the first record, identification of a first one or more records of the set of records which include the value of a first column of the plurality of columns of the first record, identification of a second one or more records of the set of records which include the value of a second column of the plurality of columns of the first record, determination of a first intersection of records between the first one or more records and the second one or more records, identification of a third one or more records of the set of records which include the value of a third column of the plurality of columns of the first record, and determination of a second intersection of records between the first intersection of records and the third one or more records.