Hierarchical Data File Transmission via Variable Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Efficient transmission of hierarchical data files over computer networks is hindered by the need to transfer unnecessary data, which increases costs and time, especially in scientific applications using formats like NetCDF and HDF, where large amounts of data are generated and require efficient storage and processing.
Innovation Solution
A method and system that modify hierarchical data files by filtering out unused variables and replacing them with pointers, using checksums to identify duplicates on the destination device, thereby reducing data transfer by only sending unique variables and optimizing storage on both the source and destination devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional file transfer methods are used to transmit hierarchical data files, then complete data transfer is ensured, but data transfer time and network bandwidth consumption increase significantly
Solution Approach 1:
The patent extracts and identifies unused variables from hierarchical data files before transmission. By analyzing the file structure and determining which variables are not accessed or needed at the destination, the system removes these unnecessary variables from the transmission set, thereby reducing transfer time while maintaining data completeness for actually needed variables
Solution Approach 2:
The patent performs preliminary analysis of the hierarchical data file structure and pre-identifies unused variables before the actual transmission process. This preliminary action includes computing checksums for all variables, comparing them against a database of known variables, and determining which variables can be safely excluded from transmission, thus optimizing the transfer process in advance
2Reliability
If all variables in hierarchical data files are transmitted, then data completeness is maintained, but network bandwidth consumption and transmission costs increase
Solution Approach 1:
The patent extracts unused variables from the hierarchical data file structure by comparing variable names and checksums against a database of expected variables. These extracted unused variables are then excluded from the transmission process, reducing network bandwidth consumption while ensuring that only necessary variables are transmitted
Solution Approach 2:
The patent changes the transmission parameters by dynamically determining which variables to include based on their usage status. By modifying the set of transmitted variables from 'all variables' to 'only used variables', the system reduces bandwidth consumption while maintaining data completeness for the actual needs
3Productivity
If unused variables are filtered out before transmission, then data transfer efficiency improves, but the complexity of the transmission system increases
Solution Approach 1:
The patent performs preliminary analysis and filtering of hierarchical data files before transmission by pre-computing checksums for all variables and comparing them against a database of known variables. This preliminary action identifies unused variables that can be excluded from transmission, improving efficiency while containing complexity through automated preprocessing
Solution Approach 2:
The transmission system performs self-service by automatically analyzing its own data files, identifying unused variables through checksum comparison, and making decisions about which variables to transmit. This self-service approach reduces the need for external complex coordination while improving transfer efficiency
4Loss of substance
If checksums are computed for all variables to identify duplicates, then data redundancy is reduced, but processing time and computational resources increase
Solution Approach 1:
The patent extracts checksum information for variables and compares it against a database of known variable checksums. By extracting and comparing only the checksum values rather than full variable contents, the system efficiently identifies duplicates and redundancy while minimizing processing time and computational resources
Data Source
AI summary
A method, a computer program product, and a computer system for transmitting a hierarchical data file from a first computer device to a second computer device. The first computer device receives a request for transmitting a first hierarchical data file. The first computer device modifies the first hierarchical data file based on a rule based filter. The first computer device receives from the second computer device a list of unique variables whose contents differ from contents in an existing file on the second computer device. The first computer device sends to the second computer device the contents of the unique variables. The contents of the unique variables are saved on the second computer device in a second hierarchical data file. The second hierarchical data file comprises pointers linking to contents of variables that are not the unique variables in the existing file.