Distributed Event Serialization with Type-Based Compression

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing database systems are ill-equipped to handle continuous data streams from applications like sensor data and financial tickers, lacking the processing flexibility required for modern event processing needs.

Innovation Solution

A distributed event processing system that serializes and processes event data against continuous queries, utilizing data compression techniques and a DAG of transformations to generate output events.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If traditional database systems are used to store and query data, then data can be stored in structured tables with SQL queries, but the system cannot handle continuous data streams effectively

Engineering Contradiction:

Improvecapability to handle continuous data streamsVSAvoidsystem architecture complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system segments the data processing architecture into distinct components: event sources generate continuous streams, event processing systems consume and transform these streams, and event sinks store the processed data. This segmentation allows each component to be optimized independently, enabling the system to handle continuous data streams while maintaining manageable complexity through modular design.

Inventive Principle:

Principle #1Segmentation

2Productivity

If data is processed in finite batches, then queries produce static results, but the system cannot provide real-time continuous processing

Engineering Contradiction:

Improvereal-time processing capabilityVSAvoidprocessing latency

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The event processing system implements continuous action by maintaining long-running query executions that process event streams indefinitely rather than in discrete batches. The system continuously consumes events from sources, applies transformations through registered functions, and outputs results to sinks in an unbroken flow, eliminating idle time between processing cycles and achieving real-time productivity with minimal latency.

Inventive Principle:

Principle #20Continuity of useful action

3Adaptability or versatility

If complex transformations are applied to event data, then processing flexibility is improved, but processing speed decreases

Engineering Contradiction:

Improveprocessing flexibilityVSAvoidevent processing speed

Core Design Contradiction:

Adaptability or versatilityVSSpeed

Solution Approach 1:

The system performs preliminary action by pre-compiling and optimizing transformation functions before they are applied to event streams. Query plans are generated in advance, and transformation logic is prepared and validated before execution. This preliminary preparation allows the system to apply complex transformations with minimal runtime overhead, maintaining both processing flexibility and high event processing speed.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS20250252107A1Data serialization in a distributed event processing system

Publication Date: 2025.08.07 ORACLE INT CORP

AI summary

A distributed event processing system is disclosed that receives a batch of events via a continuous data stream and performs the serialization of data in the batch of events. In certain embodiments, the system identifies a first data type of a first attribute for each event in a batch of events and determines a first type of data compression to be performed on data values represented by the first attribute. The system determines a first type of data compression to be performed on data values represented by the first attribute based on the first data type of the first attribute. The system then generates a first set of serialized data values for the first attribute. The system processes the first set of serialized data values against a set of one or more continuous queries to generate a first set of output events.