Encoded Data Column Mapping Without Full Decompression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face inefficiencies in processing data from multiple sources due to the need for decompression and lexical analysis before combining or performing arithmetic operations on compressed files, leading to slow data processing speeds.
Innovation Solution
A data processing method and apparatus that acquire and process encoded files in an encoded state by using encoding rules to identify and combine columns, allowing for high-speed data processing without decoding, using dynamic dictionaries to manage and adjust codes as needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If compressed files are decompressed and lexical analysis is performed before data processing, then data accuracy is ensured, but processing speed deteriorates
Solution Approach 1:
The patent applies preliminary action by performing encoding on data columns before storage and compression. The data is encoded into a standardized format that preserves structural information, allowing the data processing apparatus to identify and process specific columns directly from the compressed encoded data without full decompression. This preliminary encoding enables subsequent fast processing while maintaining data accuracy.
Solution Approach 2:
The patent extracts only the necessary encoded column data from the compressed files for processing. By using the encoding rules to identify and extract specific columns from the compressed encoded data, the system avoids the need to fully decompress and perform lexical analysis on the entire file, thus maintaining accuracy for the required data while improving processing speed.
2Productivity
If compressed files are processed without decoding, then processing speed is improved, but data processing complexity increases
Solution Approach 1:
The patent introduces encoding rules as an intermediary between the compressed data and the processing logic. The data processing apparatus uses these encoding rules to interpret and process specific columns from the compressed encoded data without full decoding. This intermediary layer simplifies the processing complexity by providing a standardized way to access and manipulate encoded data directly.
Solution Approach 2:
The patent changes the parameter representation of data by encoding it into a standardized format with specific structural properties. The encoded data uses a consistent column identification scheme that allows the processing apparatus to work directly with compressed data by interpreting encoding parameters rather than performing complex decoding and lexical analysis operations.
3Adaptability or versatility
If data is encoded in different formats from multiple data sources, then data source flexibility is maintained, but processing difficulty increases
Solution Approach 1:
The patent applies universality by creating a standardized encoding format that can represent data from multiple different data sources. The encoding rules define a universal structure that accommodates various data types and sources while maintaining consistency. This universal encoding format allows the data processing apparatus to handle data from different sources uniformly, reducing processing difficulty despite the diversity of source systems.
Solution Approach 2:
The patent transforms diverse data from multiple sources into a standardized encoded format with consistent parameters. By changing the representation parameters of incoming data to match the standardized encoding scheme, the system maintains flexibility in accepting data from various sources while simplifying subsequent processing through parameter uniformity.
Data Source
AI summary
A data processing method includes acquiring first encoded data and second encoded data when receiving a request for data processing, the first encoded data being encoded in accordance with a first encoding rule, the second encoded data being encoded in accordance with a second encoding rule, specifying a first code relating to a second code in accordance with the first encoding rule and the second encoding rule without decoding the first encoded data and the second coded data, the first code being included in the first encoded data, the second code being included in the second encoded data and generating third encoded data in which the first code and the second code are stored in association with each other.