Unified Compressed SAS Data View for Storage Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems face challenges in efficiently storing and accessing large amounts of statistical analysis data due to exponential storage requirements and delayed access, with traditional data files being interpretable by only a single statistical analysis engine, limiting data usage and accessibility.
Innovation Solution
The implementation of a unified and compressed data system that combines SAS data step views with metadata, allowing for parallel compression and decompression, enabling efficient storage and rendering of data across various systems, including non-SAS specific hosts and clients, thereby reducing storage needs and improving processing times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data files are used for storing statistical analysis data, then data can be stored in a format interpretable by a single statistical analysis engine, but storage requirements grow exponentially and data access is delayed
Solution Approach 1:
The patent segments bulk data into fixed-size data blocks that can be independently processed and stored. Each data block is a self-contained unit that can be handled separately, allowing for efficient storage management and parallel processing without requiring the entire dataset to be loaded into memory simultaneously.
Solution Approach 2:
The patent implements a nested structure where data blocks are organized within data sets, which are in turn managed by the statistical analysis system. Each data block contains nested metadata, variable information, and observation data, creating a hierarchical organization that enables efficient access and processing at multiple levels.
2Adaptability or versatility
If traditional data files are used for storing statistical analysis data, then data can be stored in a format interpretable by a single statistical analysis engine, but processing time increases due to sequential access requirements
Solution Approach 1:
The patent creates a universal data block format that can be interpreted by multiple statistical analysis engines and programming languages (SAS, R, Python, etc.). The standardized structure with metadata, variable definitions, and observation data enables different systems to process the same data blocks without conversion, facilitating parallel processing across multiple engines simultaneously.
Solution Approach 2:
The patent performs preliminary organization of data into fixed-size blocks with embedded metadata and variable information before processing. This pre-structuring allows statistical analysis engines to directly access and process specific data blocks without sequential scanning, enabling parallel processing and reducing overall processing time.
3Ease of operation
If data is stored in uncompressed format for easy access, then data retrieval is simpler, but storage space requirements increase significantly
Solution Approach 1:
The patent segments data into fixed-size blocks that can be independently compressed and stored. Each data block is a self-contained unit that can be handled separately, allowing for efficient storage management and parallel processing without requiring the entire dataset to be loaded into memory simultaneously.
Solution Approach 2:
The patent implements a nested structure where data blocks are organized within data sets, which are in turn managed by the statistical analysis system. Each data block contains nested metadata, variable information, and observation data, creating a hierarchical organization that enables efficient access and processing at multiple levels.
Data Source
AI summary
Systems and methods for compression and/or unification of statistical analysis system (SAS) data is provided. In one embodiment, a request to open a unified and compressed statistical analysis system (SAS) view file is received. The unified and compressed SAS data step view file including: an SAS data step view; compressed payload data to be used in the SAS data step view when decompressed; and a set of metadata describing characteristics of variables of the SAS data step view. Upon receiving the request, the compressed payload data is automatically decompressed, such that compressed payload data is decompressed and usable with the SAS data step view to render the SAS data step view and decompressed payload data on an electronic display of a client or host providing the request.