Partitioned Database Analytics with Dirty Flag Tracking

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data analytic systems face inefficiencies in processing and analyzing large databases, as they consume significant system resources and require unnecessary reprocessing of entire databases upon updates, leading to prolonged processing times.

Innovation Solution

The system partitions the database structure and analytics into sub-partitions and sub-analyses, allowing only updated partitions to be re-executed, with dirty flags indicating which partitions need re-processing, thereby reducing resource consumption and improving efficiency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If the entire database is re-processed upon updates, then data consistency is maintained, but processing time and resource consumption increase significantly

Engineering Contradiction:

Improvedata consistencyVSAvoidprocessing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The database is divided into multiple partitions, and the analytic operation is segmented to be applied only to specific partitions that contain updated records. This segmentation allows the system to maintain data consistency by re-processing only affected partitions rather than the entire database, thereby reducing processing time while preserving reliability.

Inventive Principle:

Principle #1Segmentation

2Reliability

If the entire database is re-processed upon updates, then accurate results are ensured, but system resource consumption increases

Engineering Contradiction:

Improveresult accuracyVSAvoidsystem resource consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The system extracts and identifies only the specific partitions that contain updated records, separating them from the rest of the database. By applying the analytic operation only to these extracted partitions rather than the entire database, the system ensures result accuracy is maintained while significantly reducing CPU, memory, and I/O resource consumption.

Inventive Principle:

Principle #2Taking out (Extraction)

3Productivity

If partition-based selective re-processing is implemented, then processing efficiency improves, but system complexity increases

Engineering Contradiction:

Improveprocessing efficiencyVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system performs preliminary actions by pre-dividing the database into partitions and pre-identifying which partitions contain updated records before executing the analytic operation. This preliminary organization and tracking of partition states enables efficient selective re-processing while keeping the added system complexity manageable through structured partition management.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS10402388B1Partition-based analytic systems and methods

Publication Date: 2019.09.03 LEVYX LLC

AI summary

A persistent distributed database system is disclosed that partitions a database structure into a plurality of partitions to optimize repeated analytics that are executed on the database structure. A repeated analytic is partitioned into sub-analytics that are executed on the database partitions, and upon repeated executions, sub-analytics are only repeated upon database partitions that have been updated since the previous execution. This drastically reduces the time to repeat execution of a repeated analytic, and frees up resources that could be better spent on other processes.