Clump Header Table and Inline Tree for Large Dataset Search

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional electronic data structures such as spreadsheets and relational databases become unwieldy and slow for large datasets, rendering high-speed search and retrieval impractical, while alternative inline tree data structures offer speed but lack flexibility for updating or adding new data.

Innovation Solution

A computer-implemented method using a clump header table, inline tree data structure, and auxiliary data structures to enable high-speed search and retrieval, allowing alteration or addition of data fields without modifying the inline tree structure, by employing a dedicated conversion program and search/filter program.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If conventional electronic data structures (spreadsheets and relational databases) are used to store large datasets, then data can be easily updated and new fields added, but search and retrieval speeds become unacceptably slow

Engineering Contradiction:

Improvesearch and retrieval speedVSAvoidease of updating data and adding fields

Core Design Contradiction:

SpeedVSEase of operation

Solution Approach 1:

The data structure is segmented into multiple tables with hierarchical levels (level 1, level 2, level 3 tables), where each table serves a specific function in the search process. This segmentation allows the system to quickly filter and retrieve data without processing the entire dataset, thereby improving search speed while maintaining operational flexibility through the modular table structure.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary data structure that converts conventional spreadsheet or database data into a specialized hierarchical format. This intermediary structure enables high-speed searching and filtering operations while the conversion process preserves the ability to update and add fields, thus resolving the contradiction between speed and ease of operation.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Quantity of substance

If the dataset size increases to accommodate more records and fields, then data comprehensiveness improves, but memory requirements and processing time increase significantly

Engineering Contradiction:

Improvedataset size and comprehensivenessVSAvoidprocessing time and memory requirements

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

By segmenting the large dataset into hierarchical tables with different levels of granularity, the system can process only the relevant portions of data needed for each search query. This reduces the effective processing time and memory requirements while maintaining access to the complete comprehensive dataset.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Different portions of the data structure are optimized for different purposes: level 1 tables are optimized for broad filtering, level 2 tables for intermediate processing, and level 3 tables for detailed retrieval. This local optimization allows the system to handle large comprehensive datasets efficiently by applying appropriate processing strategies to different data regions.

Inventive Principle:

Principle #3Local quality

3Speed

If a specialized inline tree data structure is used for high-speed searching, then search performance improves dramatically, but flexibility to update or add new data fields is lost

Engineering Contradiction:

Improvesearch performanceVSAvoidflexibility to update or add data fields

Core Design Contradiction:

SpeedVSAdaptability or versatility

Solution Approach 1:

The patent creates a dynamic conversion system that can adapt the hierarchical table structure to accommodate new data fields and update requirements. The conversion process from conventional data structures to the specialized format is designed to be flexible, allowing the data structure to evolve while maintaining high-speed search capabilities through the preserved hierarchical organization.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The hierarchical table structure is designed to serve multiple functions: it enables high-speed searching and filtering operations while also accommodating data updates and field additions through the conversion process. The universal design allows the same structure to handle both performance-critical operations and flexibility-critical operations.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS9171054B1Systems and methods for high-speed searching and filtering of large datasets

Publication Date: 2015.10.27 MOONSHADOW MOBILE INC

AI summary

A data structure comprises a clump header table, an inline tree data structure, and one or more auxiliary data structures. Each clump header record includes an indicator of a location in the inline tree data structure of corresponding binary string segments. Clump header records or auxiliary header records include indicators of corresponding locations in the corresponding auxiliary data structure. Each auxiliary data structure can be altered without necessarily altering the inline tree or clump header table. A dedicated, specifically adapted conversion program generates the clump header file, the inline tree data structure, and the one or more auxiliary data structures. The data structure can be stored on any computer-readable medium, and can be read entirely into RAM to be searched (with or without filtering on one or more filter data fields). A dedicated, specifically adapted search and filter program is employed, which can list or enumerate the retrieved data records.