Sparse Matrix Storage Using List-of-Lists and Base64 Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently encoding, communicating, and storing sparse matrices, particularly in large-scale statistical computing systems, due to the high proportion of zero elements which leads to inefficient data storage and communication, and existing formats like COO and LIL may fracture sparse matrix rows during distribution, causing computation errors and increased latency.
Innovation Solution
The proposed solution involves using a List-of-Lists format for run-length compression followed by Base64 encoding of sparse matrix rows, which allows for dynamic storage and communication of sparse matrix data, avoiding row fracturing and reducing data transmission latency by representing each row as a single character string, enabling efficient distribution across a grid-computing system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If sparse matrices are stored using conventional formats (COO, LIL), then data can be stored, but row fracturing occurs during distribution causing computation errors and increased latency
Solution Approach 1:
The patent merges the entire sparse matrix row into a single continuous data structure using List-of-Lists format, where each row is represented as one unified object containing all non-zero elements and their column indices. This prevents row fracturing during distribution across computing nodes, ensuring computational accuracy while simplifying data management.
2Quantity of substance
If sparse matrices are stored with all zero elements included, then complete data is stored, but storage efficiency and communication efficiency deteriorate
Solution Approach 1:
The patent extracts only the non-zero elements from sparse matrix rows, storing them in List-of-Lists format along with their column indices. This extraction eliminates the storage burden of numerous zero elements while preserving all meaningful data, achieving both data completeness and storage efficiency.
3Productivity
If sparse matrix rows are distributed across grid-computing devices, then parallel processing is enabled, but data transmission latency increases
Solution Approach 1:
The patent changes the data format parameter from conventional sparse formats to List-of-Lists representation, which organizes data in a more compact and transmission-friendly structure. This parameter change reduces the amount of data that needs to be transmitted during parallel distribution, thereby reducing transmission latency while maintaining parallel processing capabilities.
4Adaptability or versatility
If database systems store large sparse matrices, then analytical capabilities are enhanced, but database limitations cause storage and processing inefficiencies
Solution Approach 1:
The patent segments the sparse matrix into individual rows, with each row stored as a separate List-of-Lists structure. This segmentation allows database systems to handle large sparse matrices by processing and storing rows independently, reducing the complexity burden on the database while maintaining full analytical capabilities.
Data Source
AI summary
Methods, processes and computer-program products are disclosed for use in a parallelized computing system in which representations of large sparse matrices are efficiently encoded and communicated between grid-computing devices. A sparse matrix can be encoded and stored as a collection of character strings wherein each character string is a Base64 encoded string representing the non-zero elements of a single row of the sparse matrix. On a per-row basis, non-zero elements can be identified by column indices and error correction metadata can be included. The resultant row data can be converted to IEEE 754 8-byte representations and then encoded into Base64 characters for storage as strings. These character strings of even very large-dimensional sparse matrices can be efficiently stored in databases or communicated to grid-computing devices.