Ontology-Based Data Encoding for Sparse Field Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data encoding methods fail to optimize storage efficiency in databases by retaining values for invalid fields, such as null-value and logically deleted fields, leading to inefficient use of storage space.
Innovation Solution
Adaptive data encoding methods that utilize ontology definition data to define fields and data types, incorporating a description part and data area to store only valid field values, with optional bitmaps or lists to mark invalid fields, optimizing storage by describing the distribution and locations of valid fields based on their validity and type information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional data encoding methods are used that retain values for all fields including invalid fields, then data completeness is maintained, but storage efficiency deteriorates due to wasted space on null-value and logically deleted fields
Solution Approach 1:
The encoding structure is segmented into a description part and a data area. The description part contains validity information (bitmaps) that divide fields into valid and invalid categories, while the data area only stores values for valid fields. This segmentation allows the system to maintain data completeness through the description part while improving storage efficiency by excluding invalid field values from the data area.
Solution Approach 2:
Invalid field information (null-value fields and logically deleted fields) is extracted and removed from the data storage process. The validity bitmap identifies which fields should be excluded, and only the remaining valid fields are encoded into the data area, thereby eliminating wasted storage space while preserving the ability to reconstruct complete data through the description part.
2Volume of stationary object
If encoding methods store values for all fields to ensure data completeness, then information integrity is maintained, but storage space consumption increases
Solution Approach 1:
A validity bitmap serves as an intermediary between the field definitions and the stored data values. This bitmap acts as a marker that indicates which fields contain valid data and which are invalid (null or logically deleted). During decoding, the bitmap guides the reconstruction process, ensuring data integrity is maintained while avoiding storage of invalid field values, thus optimizing storage space.
3Device complexity
If the encoding structure includes detailed description of all fields including invalid ones, then field information completeness is maintained, but encoding complexity increases
Solution Approach 1:
The encoding structure dynamically adapts based on the validity status of fields. Rather than using a static structure that describes all fields uniformly, the system uses a bitmap to dynamically indicate which fields are valid and should be described in the data area. This dynamic approach reduces encoding structure complexity by eliminating the need to describe invalid fields while preserving complete field information through the combination of bitmap and selective data storage.
Data Source
AI summary
This specification disclose methods, apparatus, devices, and systems for encoding and decoding data. In one implementation, a method includes: obtaining instance data, and generating, based on ontology definition data, encoded data corresponding to the instance data. The encoded data comprise a data area storing valid fields of the instance data, and a description part. The description part is configured to describe a distribution of the valid field by using validity information of the fields when a quantity of the invalid fields is greater than or equal to a first threshold. The description part is configured to describe a distribution of the valid field by using at least one of size information or type information of the fields when the quantity of the plurality of invalid fields is smaller than the first threshold.