Data Management Device for Non-Structural Log Storage

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional data management systems face challenges in efficiently handling non-structural data, particularly in column-oriented data structures, where data structuring is difficult, and managing indexes is cumbersome, leading to high processing times and limitations in adding or deleting data, especially when dealing with large amounts of non-structural 'logs' stored in formats like JSON or XML.

Innovation Solution

A data management device and method that includes an interpreting unit to convert input records into abstract expressions and a converting unit to store data sets with associated index information as column data, allowing for efficient storage and retrieval of data without the need for dimensional compression, enabling direct access to specific data items without expanding all columns.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data is stored using column-oriented data structure, then statistical processing efficiency is improved, but data addition and deletion operations become difficult

Engineering Contradiction:

Improvestatistical processing efficiencyVSAvoiddata addition and deletion

Core Design Contradiction:

ProductivityVSEase of operation

Solution Approach 1:

The patent segments data storage into two distinct structures: row-oriented storage for efficient data addition and deletion operations, and column-oriented storage for efficient statistical processing. The conversion unit transforms data between these two formats, allowing each structure to be optimized for its specific purpose without compromise.

Inventive Principle:

Principle #1Segmentation

2Ease of manufacture

If row-oriented data structure is used with JSON function, then automatic data tree structure generation is achieved, but network cost, storage cost, and processing time increase

Engineering Contradiction:

Improveautomatic data tree structure generationVSAvoidprocessing time

Core Design Contradiction:

Ease of manufactureVSLoss of time

Solution Approach 1:

The patent extracts the data tree structure generation function from the storage layer and implements it in the conversion unit. This allows automatic structure generation to occur during data transformation from row-oriented to column-oriented format, eliminating the need for continuous JSON processing and reducing ongoing network and processing costs.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent performs data structure conversion and tree structure generation in advance during the data ingestion phase, rather than processing it repeatedly during queries. This preliminary action reduces subsequent processing time and computational overhead significantly.

Inventive Principle:

Principle #10Preliminary action

3Productivity

If column-oriented system is used, then statistical processing is efficient, but managing all indexes and handling write failures becomes difficult

Engineering Contradiction:

Improvestatistical processing efficiencyVSAvoidindex management complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent introduces a row-oriented storage layer as an intermediary between data input and column-oriented statistical processing. This intermediary handles complex operations like index management and write failure handling in the flexible row-oriented format, while presenting simplified data to the column-oriented storage for efficient statistical queries.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS11487729B2Data management device, data management method, and non-transitory computer readable storage medium

Publication Date: 2022.11.01 YAHOO JAPAN CORP

AI summary

A data management device is provided, which includes an interpreting unit that interprets an input record, and convert the input record to abstract expression by which a correspondence relation between a data item and data itself is recognizable; and a converting unit that stores, in a storage unit, a data set, in which the data itself and index information capable of identifying the record are associated, as column data for each of the data items.