SPICE Tree Data Structure for Document Chunking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data structures for collaboration applications require entire versions of documents to be downloaded or uploaded, leading to high storage and bandwidth costs, as well as increased latency and processor requirements.
Innovation Solution
The implementation of a Spatially Partitioned Ideally Chunked Entity (SPICE) tree data structure, which splits data into chunks and allows for efficient navigation between related chunks, reducing the need to download or upload entire documents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If entire versions of documents are downloaded or uploaded using current data structures, then data integrity and completeness are maintained, but storage costs and bandwidth requirements increase significantly
Solution Approach 1:
The patent segments the document data structure into multiple chunks, where each chunk represents a portion of the document. This allows clients to download and access only the specific chunks they need rather than the entire document, significantly reducing bandwidth consumption while maintaining data integrity through proper chunk management and referencing.
2Quantity of substance
If entire versions of documents are downloaded or uploaded, then complete data is available for processing, but processor requirements and latency increase
Solution Approach 1:
By dividing the document into discrete chunks that can be independently processed, the system enables selective processing of only the necessary data portions. This reduces the processing burden on clients and servers, lowering latency and improving overall system productivity while maintaining complete data availability when needed.
3Ease of operation
If B-tree data structures are used to break down data into sub-categories, then data organization improves, but sub-categories still must be downloaded in their entirety
Solution Approach 1:
The patent extends the segmentation concept beyond traditional B-tree structures by creating independent, addressable chunks that can be selectively accessed. Each chunk is a self-contained unit with references to other chunks, allowing clients to navigate and download only the specific segments needed for their operations, eliminating the requirement to download entire sub-categories.
4Ease of operation
If binary blobs such as bitmaps are uploaded and downloaded separately, then individual data elements can be accessed, but the underlying data structures have no incrementality and do not scale well
Solution Approach 1:
The patent implements a dynamic chunking system where the number and size of chunks can adapt based on document size and access patterns. The chunked structure with embedded references enables the system to scale efficiently as documents grow, allowing clients to access only the necessary chunks without requiring complete data structure reDownloads, thereby achieving both efficient data access and scalability.
Data Source
AI summary
Systems and methods for providing a Spatially Partitioned Ideally Chunked Entity (“SPICE”) tree data structure are provided herein. In an example, a computerized method for using a SPICE tree includes determining a document defined by a SPICE tree data structure and navigating to content within the document based on the SPICE tree data structure. The SPICE tree data structure includes a root chunk containing radar nodes and a plurality of chunks. Each chunk includes one or more object nodes, each of which corresponds to a document attribute. Each of the radar nodes includes a reference to one of the chunks and the object nodes include a placeholder node that provides a reference to another chunk based on a position of the placeholder node within the respective chunk. The root chunk and the chunks are arranged in a hierarchical arrangement with the root chunk being a parent to the chunks.