Deterministic incremental sliding-window analytics engine using distributed remote in-memory cache with multi-tenant isolation for high-speed real-time analytics and security event processing

Abstract

"DETERMINISTIC INCREMENTAL SLIDING-WINDOW ANALYTICS ENGINE USING DISTRIBUTED REMOTE IN-MEMORY CACHE WITH MULTI-TENANT ISOLATION FOR HIGH-SPEED REAL-TIME ANALYTICS AND SECURITY EVENT PROCESSING" The disclosed invention introduces a deterministic engine for incremental sliding-window analytics that employs a distributed remote in-memory cache to achieve rapid real-time processing with multi-tenant isolation, particularly tailored for analytics and security event handling. The present invention also provides for a computer-implemented method for deterministic sliding- window statistical aggregation over streaming data, the method comprising the steps of: generating a unique aggregation key for the streaming data; acquiring an atomic lock on the aggregation key in a remote in-memory cache; storing, in the remote in-memory cache, a timestamped value associated with the aggregation key; computing a window boundary time based on a current timestamp and a predefined window duration; identifying expired entries in the remote in-memory cache that have timestamps earlier than the window boundary time; accumulating a delta value from the expired entries and applying a cumulative subtraction of the delta value to remaining non-expired entries; computing at least one statistical metric based on the non-expired entries after the cumulative subtraction; and releasing the atomic lock on the aggregation key.

Description

[0001] TITLE OF THE INVENTION

[0002] DETERMINISTIC INCREMENTAL SLIDING-WINDOW ANALYTICS ENGINE USING DISTRIBUTED REMOTE IN-MEMORY CACHE WITH MULTI-TENANT ISOLATION FOR HIGH-SPEED REAL-TIME ANALYTICS AND SECURITY EVENT PROCESSING

[0003] CROSS-REFERENCE TO RELATED APPLICATIONS

[0004] This patent application asserts priority from and incorporates by reference the entirety of the Indian Provisional Patent Application entitled "HIGHSPEED REAL-TIME ANALYTICS FOR MOVING TIME WINDOW USING REMOTE CACHE," which was submitted on February 26, 2025, by the inventors.

[0005] FIELD OF THE INVENTION

[0006] The present disclosure pertains to the domain of data processing systems, with a specific focus on engines designed for real-time analytics over streaming data. More particularly, it addresses mechanisms for executing deterministic incremental aggregations within sliding time windows, leveraging distributed in-memory caching technologies to ensure high-speed performance, concurrency safety, and isolation across multiple tenants, especially in contexts involving security event monitoring and anomaly identification.

[0007] BACKGROUND OF THE INVENTION

[0008] In contemporary data-intensive environments, particularly those involving event-driven architectures like security information and event management (SIEM) platforms, there is a continuous influx of high-velocity data streams that necessitate immediate aggregation and analysis within defined temporal boundaries to facilitate prompt decision-making and the detection of irregularities. Conventional approaches, such as those relying on relational or time-series databases, often depend on persistent storage mechanismsthat entail range queries, index upkeep, and complete recomputations of window contents during each update, resulting in substantial delays when handling elevated throughput levels. These traditional systems frequently encounter issues related to concurrent access conflicts, unpredictable behaviour under simultaneous operations, suboptimal removal of outdated information, and linear computational expenses proportional to the entire window size, rendering them inadequate for scalable distributed settings.

[0009] Within SIEM frameworks, these limitations are intensified by requirements for aggregations centered on specific entities, instantaneous threshold-based anomaly alerts, temporal correlations across diverse events, mechanisms to avoid redundant notifications, and segregation of data among different organizational tenants. The processing of voluminous security logs — originating from activities such as user authentications, network interactions, firewall activities, device telemetry, application executions, domain name resolutions, resource accesses, and privilege elevations — demands reliable and thread-safe handling to prevent erroneous detections or overlooked risks.

[0010] Existing technologies have sought to mitigate certain elements of these challenges, yet they do not offer a holistic, predictable, and expandable framework for progressive sliding-window analytics through distributed inmemory storage.

[0011] For example, one prior solution, as detailed in U.S. Patent No. 10,698,829 B2, outlines a technique for transferring data directly between hosts in localized caches within virtualized infrastructures, where historical hosting records track prior locations of data objects before relocation, and retrieval operations first consult local and designated host caches prior to accessing a shared repository. This method emphasizes cache transitions and retrieval in virtual setups but overlooks aspects like time-bound window aggregations on flowing data, timestamp-driven boundary determinations, or incremental adjustments to circumvent exhaustive recalculations in scenarios with rapid data ingress.Another example, found in U.S. Patent No. 9,122,786 B2, presents frameworks for conducting statistical examinations of extensive, varied datasets via complex event processing methodologies, wherein operators for stream analysis subscribe to cache notifications, approximate density functions for attribute distributions, and integrate validity assessments with these distributions to guide optimizations such as cache tuning. Although this facilitates the evaluation of cache items and their temporal relevance, it omits provisions for atomic isolation of keys, sharded distribution via uniform hashing, or targeted adaptations for instantaneous security log handling with tenant-specific segregation, potentially leading to suboptimal performance in assured sliding-window operations.

[0012] Further, U.S. Patent No. 9,021,021 B2 describes a system for monitoring and analyzing usage in mobile networks, which assesses the effectiveness of traffic optimizations, produces efficiency reports, and implements reduction strategies quantified by optimization occurrences, encompassing details on applications and user behaviours. This emphasizes network management and reporting but neglects incremental aggregation strategies employing inmemory caches for time-sliding windows, as well as techniques for subtracting accumulated deltas from expired items or calculating metrics with complexity limited to the count of expired elements.

[0013] Additionally, U.S. Patent No. 9,141,723 B2 proposes caching approaches for data tied to sliding windows in computational systems, concentrating on maintenance paradigms, but it mainly covers broad caching without embedding transactional monitoring for concurrent integrity or scaled sharding customized for SIEM deployments.

[0014] In U.S. Patent Application Publication No. 2022 / 0019497 Al, a system aggregates alerts using variable-length sliding windows spanning from brief intervals to extended periods, with iterative pattern extraction to derive rules from alert records; however, this is confined to alert consolidation and lacks support for cumulative delta adjustments or namespace-based tenant separation.Moreover, U.S. Patent Application Publication No. 2012 / 0072656 Al elucidates multi-level caching with indexed structures across tiers, storing key-value associations as segmented lists, utilizing logs, allocation managers, and zone controllers for governance, alongside merging processes that shift ordered lists between levels while preserving high-access segments in superior tiers. This enhances storage hierarchy efficiency but does not encompass the storage of time-stamped entries, boundary time derivations, or refinements for real-time computations of aggregates such as totals, means, tallies, extremes, spans, variation rates, central tendencies, or predominant values in dynamic streams.

[0015] Turning to non-patent sources, documentation for Redis and associated guides on enacting sliding-window constraints for rate control via ordered sets and scripting languages (as seen in resources like redis. io / tutorials / develop / dotnet / aspnetcore / rate-limiting / sliding- window) illustrate practical deployments of in-memory stores for accurate flow regulation through models like counters or token systems, capitalizing on swift retrieval and alteration capabilities. Nevertheless, these are predominantly oriented toward limitation enforcement rather than full-fledged analytics platforms incorporating resilient replication, recovery from failures, or entity-focused irregularity spotting in security contexts.

[0016] Academic contributions, including the paper "Low-Latency Sliding-Window Aggregation in Worst-Case Constant Time" from the DEBS 2017 conference, propose methods like the DABA algorithm that attain constant-time aggregation for first-in-first-out windows by partitioning inputs and synchronizing scans, suitable for combinable operations, but these do not scale to distributed in-memory setups with multi-tenant capabilities or practical SIEM integrations.

[0017] Comparable open-source implementations of window aggregation routines offer operator-based techniques, and community discussions on functional approaches to sliding windows underscore efficient scan pairings, yet they fall short in fusing with enterprise features such as hashing for distributionor tenant quotas. Further, articles such as "Sliding Window Based Rate Limiter" endorse in-memory solutions like Redis for operational use, highlighting benefits in expiration handling, atomic operations, locking, and sorted collections, which underscore the value of memory-based storage for immediate processing in constrained environments.

[0018] A NPL discloses "Slider: Incremental Sliding-Window Computations for Large-Scale Data Analysis" by Pramod Bhatotia et aL): Publicly disclosed before 2024 (e.g., ACM Middleware 2014), this is prior art under § 102(a)(1). Slider enables incremental sliding-window analytics over streams using a distributed in-memory cache for memoization (fast reuse of subcomputations). It maintains dependency graphs for change propagation, updating outputs incrementally as windows slide (adding new data, dropping old). This mirrors the invention's incremental approach and inmemory caching for scalability.

[0019] However, Slider is general-purpose, lacks multi-tenant isolation, deterministic locking, delta reduction specifics, or security event focus. It's cited in later patents (e.g., US 10,163,061 B2, 2018, on analytic retraining), but no direct patent on Slider itself was found, suggesting it's in the public domain. This could render basic incremental sliding-window claims obvious, but the invention's integration with multi-tenancy and security thresholds adds potential novelty.

[0020] SUMMARY OF THE INVENTION

[0021] The disclosed invention introduces a deterministic engine for incremental sliding-window analytics that employs a distributed remote in-memory cache to achieve rapid real-time processing with multi-tenant isolation, particularly tailored for analytics and security event handling. This system overcomes the shortcomings of prior arts by integrating atomic locking for concurrency control, delta-based reductions to minimize recomputation efforts, sharded architecture for scalability, and namespace segregation for tenant privacy. Through these mechanisms, the engine ensures predictablebehaviour, efficient cleanup of expired data, and computation of statistical metrics over streaming inputs, enabling applications in high-stakes domains like SIEM where timely insights are paramount.

[0022] Therefore such as herein described there is provided a computer-implemented method for deterministic sliding-window statistical aggregation over streaming data, the method comprising the steps of: generating a unique aggregation key for the streaming data; acquiring an atomic lock on the aggregation key in a remote in-memory cache; storing, in the remote inmemory cache, a timestamped value associated with the aggregation key; computing a window boundary time based on a current timestamp and a predefined window duration; identifying expired entries in the remote inmemory cache that have timestamps earlier than the window boundary time; accumulating a delta value from the expired entries and applying a cumulative subtraction of the delta value to remaining non-expired entries; computing at least one statistical metric based on the non-expired entries after the cumulative subtraction; and releasing the atomic lock on the aggregation key.

[0023] Also herein described is a distributed system for deterministic sliding-window statistical aggregation over streaming data, configured to execute the aforesaid method steps, the system comprising of an ingestion module configured to receive the streaming data; an aggregation engine configured to perform aggregations using aggregation keys; a shard router configured to distribute the aggregation keys across cache nodes using consistent hashing; a tenant isolation manager configured to enforce data separation using tenant identifiers and namespace separation; and a distributed remote in-memory cache cluster comprising the cache nodes.

[0024] In another embodiment, there is also provided a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the aforesaid method steps.BRIEF DESCRIPTION OF THE ACCOMPANYING DRAWINGS

[0025] The accompanying drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles thereof. These figures are not necessarily to scale and are provided for clarity in understanding the disclosed concepts.

[0026] FIG. 1 illustrates the detailed component architecture of the deterministic sliding time window engine, depicting the flow from the aggregation key manager through atomic lock management, timestamp handling, window boundary calculation, store adaptation, delta reduction, window entry list management, statistics calculation, and threshold evaluation, integrated with a distributed remote in-memory cache cluster that supports sharding, replication, failover recovery, and horizontal scalability.

[0027] FIG. 2 shows the analytics engine component diagram, outlining the interactions among the event ingestion module, aggregation engine, key isolation engine, sliding window manager, delta reduction engine, statistical computation engine, distributed cache cluster, tenant isolation manager, shard router, result publisher, and security alert evaluation module.

[0028] FIG. 3 depicts the sliding window sequence flow, illustrating the step-by-step process of key generation, lock acquisition, timestamp capture, boundary computation, expired entry identification, delta accumulation, entry removal, cumulative subtraction, new value appendage, statistical computation, and lock release.

[0029] FIG. 4 presents the delta reduction activity flow, focusing on how expired values are subtracted cumulatively from subsequent entries to preserve aggregation correctness without full re-computation.

[0030] FIG. 5 illustrates the aggregation key lifecycle state diagram, showing states from key generation and acquisition to isolation, update, release, and potential shard migration or replication.FIG. 6 shows the distributed sharded cache architecture, highlighting consistent hashing for key assignment, independent shard execution of window logic, replication for continuity, and cluster rebalancing via shard migration.

[0031] FIG. 7 depicts the multi-tenant isolation architecture, with tenant-specific namespaces, configurations, thresholds, quotas, and access controls.

[0032] FIG. 8 illustrates the enterprise deployment topology, including ingestion modules, aggregation engines, shard routers, tenant managers, and the cache cluster in a fault-tolerant setup.

[0033] DETAILED DESCRIPTION

[0034] The invention provides a comprehensive system and method for conducting deterministic incremental aggregations within sliding time windows over streaming data, utilizing a distributed remote in-memory cache to deliver high-speed real-time analytics while maintaining multi-tenant isolation. This approach is particularly advantageous for processing security events, where rapid and accurate insights are essential to identify potential threats without delays or inconsistencies. As illustrated in FIG. 1, which presents the Deterministic Sliding TimeWindow Engine - Detailed Component Architecture, the engine's core functionality is structured around the TimeWindow Engine Core, serving as the central hub that orchestrates all operations. The Aggregation Key Manager, positioned at the top of this architecture, generates unique keys by incorporating tenant identifiers, such as "tenantA_userl23" for a user-specific aggregation in Tenant A, ensuring namespace isolation and supporting multi-tenant environments where different organizations process data independently without interference. For example, in a SIEM embodiment, this manager might build a key like "orgl_ip_192. 168.1.1" for tracking network activity from a specific IP in organization 1, preventing data leakage to organization 2.

[0035] This component connects downward to the Atomic Lock Manager, which employs transactional watches to acquire locks on keys atomically, attachprecise timestamps from the Timestamp Manager, and guarantee isolation during updates. In practice, if two concurrent events arrive for the same key, such as simultaneous authentication logs for a user, the lock ensures only one proceeds, avoiding race conditions that could lead to inaccurate counts; the rejected transaction retries automatically, maintaining determinism. The Timestamp Manager supplies the current system time, synchronized across nodes to prevent drift in distributed setups, ensuring boundary calculations are consistent even in geo-redundant clusters.

[0036] Continuing with FIG. 1, the Window Boundary Calculator receives the window duration — configurable per tenant, say 5 minutes for real-time anomaly detection — and computes the boundary as current time minus duration, defining a sliding cutoff like "entries before 2026-02-19 10:55:00" for a 10:00:00 window start. This deterministic boundary enables precise expiration checks, contrasting with probabilistic methods in prior art that might introduce variability. The Window Store Adapter acts as the interface to the Distributed Remote In-Memory Cache Cluster, handling operations like fetching serialized entry lists (e.g., a list of [timestamp: value] pairs sorted chronologically) and updating them atomically. In an embodiment for financial transaction monitoring, this adapter might retrieve a list of trade volumes over the last hour, de-serializing them for processing. The Delta Reduction Engine, a key innovation, identifies expired entries (those with timestamps below the boundary), accumulates their values into a delta — for instance, summing expired authentication failures — and subtracts this delta cumulatively from remaining entries to preserve aggregate correctness without full re-computation.

[0037] Consider an example where a window holds failures at tl:2, t2:3, t3: 1; if tl expires, the delta (2) is subtracted from t2 (now 1) and t3 (now -1, but adjusted for metric type), ensuring O(n_expired) complexity, which is efficient when expirations are few, as in steady-state streams.

[0038] The Window Entry List maintains these timestamped values in sorted order, appending new ones like [current_timestamp: new_value], facilitating quickboundary traversals. The Statistics Calculator then uses this cleaned list to compute metrics; for sum, it totals values; for average, divides by count; for rate-of-change, compares current sum to prior snapshots; for median, sorts and selects middle; for mode, identifies most frequent. In a security example, it might calculate the average login attempts per minute over a 10-minute window, flagging if exceeding 5. The Threshold Evaluator compares these metrics against tenant-specific thresholds, such as static (e.g., >10 failures), dynamic (e.g., >2x rolling average), or percentile-based (e.g., >95th percentile of historical rates), triggering alerts if breached. The cache cluster itself, shown with notes on sharding (consistent hashing assigns keys to shards, e.g., key hash mod shard count), replication (each shard has primaries and replicas for failover, syncing via leader-follower), failover recovery (automatic promotion of replicas on primary failure), and scalability (adding shards redistributes keys minimally), ensures high availability; in an loT embodiment, this handles millions of sensor readings without bottlenecks.

[0039] Referring to FIG. 2, the Analytics Engine Component Diagram provides a high-level hierarchical view, starting with the Event Ingestion Interface at the apex, which accepts streaming data from sources like endpoint telemetry or network logs, buffering them for processing to handle bursts, such as a DDoS attempt generating thousands of connection events per second. This interface routes events to the Aggregation Key Manager, which derives keys from event attributes — e.g., extracting user ID from an authentication log — and branches to the Sliding Window Manager for window-related logic and the Atomic Lock Manager for concurrency control.

[0040] The Sliding Window Manager oversees duration resolution and feeds into the Window Boundary Calculator, which, as in FIG. 1, computes boundaries; in an embodiment for network monitoring, a 1-hour window might track bytes transferred, expiring old data to focus on recent anomalies. The Atomic Lock Manager ensures thread -safety, linking to the Delta Reduction Engine. This engine flows to the Statistics Calculator for metric computation, then to the Threshold Evaluator for anomaly checks — e.g., if outbound connectionsexceed 100 in 5 minutes, flag as potential exfiltration — and culminates in the Result Publisher, which disseminates results via APIs or alerts. The Distributed Remote In-Memory Cache, depicted as an oval connected via arrows, serves as the persistent yet fast storage, interacting bidirectionally for reads / writes; in a multi-tenant SIEM example, it stores separated data for tenants, enabling isolated analytics.

[0041] FIG. 3 elaborates on the Sliding Window Sequence Flow, a sequence diagram emphasizing temporal interactions. Initiated by a Caller (e.g., an ingestion node), the calculate(key, window, value) request reaches the Analytics Engine, which signals the Lock Manager to watch(key), establishing a transactional context. The engine then fetches entries from the Cache Cluster, perhaps retrieving a list of 50 timestamped values for a busy user. It requests boundary computation from the Boundary Calculator, receiving the cutoff time. Expired entries are identified and listed by the Delta Engine, which returns this list for removal in the cache.

[0042] The engine appends the new value, computes the metric via the Statistics Calculator — e.g., sum of failures now at 15 — and evaluates thresholds with the Threshold Evaluator, potentially triggering an alert if >10. Finally, it unwatches the key and returns the result. In an embodiment for DNS query analysis, this sequence might correlate high query rates with malware indicators, with arrows showing synchronous calls for determinism; asynchronous variants could be considered for lower-latency but are avoided here to ensure order.

[0043] As shown in FIG. 4, the Delta Reduction Activity Flow flowchart details the procedural steps for handling updates. Starting at the top circle, a new timestamped value arrives — say, a firewall log at t= 10:01 with value=l block. Existing entries are fetched (e.g., prior blocks at t=9:55:2, t=9:58:3, t= 10:00:1 for a 5-minute window). Boundary time is computed (10:01 - 5min = 9:56). A diamond decision checks for expirations (yes, t=9:55 < 9:56); if yes, identify (t=9:55), accumulate delta (2), remove it, and apply subtraction to later entries (t=9:58 becomes 1, t= 10:00 becomes -1 adjusted per metric).If no, skip to append new value, compute metric (e.g., sum=3), and return. In a process execution embodiment, this flow efficiently prunes old launches, computing frequency rates; edge cases like empty windows (initial append) or all expirations (reset to new value) are handled gracefully, with the flowchart's paths covering concurrent-free updates due to upstream locking.

[0044] FIG. 5's Aggregation Key Lifecycle State Diagram traces key states from Idle, transitioning via watch (key) to Locked, preventing others from modifying. Fetching retrieves entries, Evaluating computes boundaries and checks expirations; if present, Cleaning accumulates / applies delta and removes, with "delta applied" leading to Updating (append value). Computing calculates metrics, then Releasing unwatches back to Idle. If no expirations, direct to Updating; "unwatch(key)" aborts to Idle on failures. In an example for privilege escalations, a key might cycle rapidly during attacks, with states ensuring no lost events; nuances include timeout on locks to avoid deadlocks, and recovery from partial states via cache consistency.

[0045] In FIG. 6, the Distributed Sharded Cache Architecture shows Analytics Nodes routing via a Shard Router with consistent hashing — e.g., hashing "userl23" to Shard 2's Primary 2 and Replica 2. The Cache Cluster has shards like Shard 1 (Primary 1, Replica 1), enabling load distribution; in failover, Replica 1 promotes to primary. For a large-scale embodiment in cloud monitoring, this allows handling petabytes by adding shards, with minimal reshuffling.

[0046] FIG. 7's Multi-Tenant Isolation Architecture separates Tenant A (with Threshold Config A like >5 failures, Aggregation Keys A, Namespace A) from Tenant B, both feeding into the Distributed Cache Cluster. Isolation via key prefixing (e.g., "A_key"), logical namespaces (virtual partitions), memory quotas (e.g., 1GB per tenant), and independent thresholds prevents crosstalk; in an example, Tenant A detects user anomalies while Tenant B monitors hosts, without shared visibility.Finally, FIG. 8 illustrates the Enterprise Deployment Topology, with Client Event Sources (e.g., Endpoint Telemetry sending process data, Network Logs for connections, Authentication Logs for logins) entering an Ingestion Cluster via Load Balancer to Ingestion Nodes for preprocessing. This flows to the Analytics Cluster's TimeWindow Engine Nodes for aggregation, connected to the Distributed Cache Cluster's Sharded Cache Nodes. Outputs go to Alerting & API Layer's Analytics API (for queries) and Alert Publisher (for notifications). In a full SIEM deployment, this topology scales horizontally, handling edge cases like node failures via replication, ensuring real-time threat detection across diverse sources.

[0047] The engine's core functionality revolves around generating unique aggregation keys that incorporate tenant identifiers to ensure data separation, acquiring atomic locks via transactional watches to prevent race conditions during concurrent accesses, capturing current timestamps to define window boundaries precisely, and storing timestamped values associated with each key in the cache for quick retrieval and manipulation.

[0048] Upon receiving a new value for aggregation, the system computes the window boundary by subtracting the specified duration from the current timestamp, thereby establishing a clear cutoff for data relevance. It then retrieves all existing timestamped entries from the cache linked to the aggregation key and identifies those that fall outside the boundary, marking them as expired. Rather than rescanning and re-computing the entire window contents — which would impose unnecessary computational overhead — the engine accumulates the values of these expired entries into a delta and performs a cumulative subtraction from the remaining valid entries, preserving the integrity of ongoing aggregations while efficiently pruning outdated data. This delta reduction mechanism ensures that cleanup operations are confined to a complexity proportional only to the number of expired items, typically much smaller than the full window size, thus optimizing performance in high-throughput environments.Following the removal of expired entries and the application of the delta adjustment, the new timestamped value is appended to the window's entry list in the cache, maintaining chronological order for subsequent operations. The system then proceeds to compute one or more statistical metrics based on the updated set of valid entries, supporting a range of functions including summation, averaging, counting, determination of minimum and maximum values, calculation of ranges, assessment of rate-of-change, identification of medians, or modes. These metrics are derived directly from the cleaned window, leveraging the pre-adjusted values to avoid redundant calculations. Once the metrics are generated, the atomic lock on the key is released, allowing other processes to access it safely, and the results are returned or published as needed, completing the incremental update cycle.

[0049] To support scalability and fault tolerance, the aggregation keys are distributed across a sharded cache cluster using consistent hashing, which deterministically assigns keys to specific shards based on their hash values, enabling horizontal expansion by adding more nodes without disrupting existing distributions. Each shard operates independently to execute the sliding-window logic, processing its allocated keys in isolation to distribute workload evenly. Replication within shards ensures data redundancy, with primary and replica instances synchronizing to provide continuity during failovers, and mechanisms for shard migration facilitate dynamic rebalancing in response to changing loads or node failures. The cache cluster also incorporates checkpoint snapshots and state recovery protocols to restore aggregation states after interruptions, minimizing data loss and downtime in production settings.

[0050] Multi-tenant isolation is achieved by embedding tenant-specific identifiers into aggregation keys and employing namespace separation within the cache, logically partitioning data to prevent cross-tenant access or interference. Each tenant can configure independent sliding-window parameters, such as durations and metric types, alongside dedicated thresholds for anomaly detection, memory allocations to control resource usage, and rate limits to manage ingestion paces. This setup allows fortailored deployments where organizations share the underlying infrastructure but maintain complete data sovereignty, with optional cryptographic enhancements for sensitive environments to further enforce access controls.

[0051] In embodiments focused on security event processing, the streaming data consists of structured logs from various sources, including authentication attempts, network connections, firewall interactions, endpoint monitoring, process launches, DNS resolutions, file operations, and privilege changes. Aggregation keys are constructed from entity attributes like user identifiers, host names, IP addresses, process IDs, session tokens, or rule identifiers, enabling entity-centric analysis. For instance, the engine can track authentication failure counts per user within a sliding window, monitoring for patterns indicative of brute-force attacks, or measure outbound connection frequencies per host to detect exfiltration attempts. Similarly, it assesses process execution rates on endpoints, data volumes transferred via IPs, or query rates for domains, providing granular insights into potential security incidents.

[0052] The computed metrics are evaluated against configurable thresholds, which may be static values, dynamic rolling averages, deviation measures, or percentile-based benchmarks, to determine if anomalies warrant action. When a metric exceeds its threshold, the system generates a security alert signal, which can trigger notifications or automated responses. To prevent alert fatigue, duplicate triggers within the same window are suppressed, with suppression states resetting automatically upon window expiration. Furthermore, the engine supports cross-event correlations by associating multiple event types with common entities within the window, such as linking failed logins to subsequent privilege escalations or process starts to network outflows, enhancing threat detection through contextual awareness.

[0053] The overall architecture integrates an ingestion module for handling incoming events, an aggregation engine for performing the windowoperations, a shard router for directing traffic, a tenant isolation manager for enforcing separations, and the distributed cache cluster as the backbone for storage. An alerting and API layer provides interfaces for querying results and publishing alerts, while the system as a whole is deployable in enterprise topologies with load-balanced ingestion nodes, analytics clusters, and sharded caches to manage large-scale operations.

[0054] INDUSTRIAL APPLICABILITY

[0055] This invention finds practical utility in fields requiring real-time data processing, such as security information and event management systems for threat monitoring, enterprise-level log analysis for operational insights, cloud-hosted security oversight for distributed environments, anomaly detection platforms for identifying deviations, network telemetry supervision for performance tracking, financial transaction streams for fraud prevention, and Internet of Things analytics for device behaviour assessment.

[0056] Examples of workability include the validated results where the framework outperforms data-driven baselines, reducing errors while maintaining efficiency.

[0057] While the invention has been described with reference to specific embodiments, those skilled in the art may contemplate many modifications without departing from the scope. The description is illustrative and not limiting.

Claims

We Claim-1. A computer-implemented method for deterministic sliding-window statistical aggregation over streaming data, the method comprising the steps of:generating a unique aggregation key for the streaming data; acquiring an atomic lock on the aggregation key in a remote inmemory cache;storing, in the remote in-memory cache, a timestamped value associated with the aggregation key;computing a window boundary time based on a current timestamp and a predefined window duration;identifying expired entries in the remote in-memory cache that have timestamps earlier than the window boundary time;accumulating a delta value from the expired entries and applying a cumulative subtraction of the delta value to remaining non-expired entries;computing at least one statistical metric based on the non-expired entries after the cumulative subtraction; andreleasing the atomic lock on the aggregation key.

2. The method of claim 1, wherein identifying and removing the expired entries is performed with a time complexity of O(n_expired), where n_expired is the number of expired entries.

3. The method of claim 1, wherein applying the cumulative subtraction of the delta value avoids recomputation of statistical metrics over an entire set of entries within the sliding window.

4. The method of claim 1, wherein the at least one statistical metric is selected from the group consisting of sum, average, count, minimum, maximum, range, rate-of-change, median, and mode.

5. The method of claim 1, wherein the remote in-memory cache is implemented as a distributed cluster of cache nodes.

6. The method of claim 5, wherein the aggregation key is assigned to one of the cache nodes using consistent hashing for sharding.

7. The method of claim 5, wherein the distributed cluster includes data replication across the cache nodes to provide failover continuity and data recovery.

8. The method of claim 1, wherein the aggregation key includes a tenant identifier to distinguish data associated with different tenants.

9. The method of claim 8, wherein data for each tenant is isolated in the remote in-memory cache using namespace separation, independent window configurations, and per-tenant memory quotas.

10. The method of claim 1, wherein the streaming data comprises structured security event logs selected from the group consisting of authentication events, network connection events, firewall logs, endpoint telemetry, process execution events, DNS query logs, file access events, and privilege escalation events.

11. The method of claim 10, wherein the aggregation key is derived from at least one security entity selected from the group consisting of a user identifier, a host identifier, an IP address, a process identifier, and a session identifier.

12. The method of claim 10, wherein the at least one statistical metric comprises a count of authentication failures associated with a security entity within the sliding window.

13. The method of claim 10, wherein the at least one statistical metric comprises a frequency of network connections associated with a security entity within the sliding window.

14. The method of claim 10, wherein the at least one statistical metric comprises a frequency of process executions associated with a security entity within the sliding window.

15. The method of claim 10, further comprising comparing the at least one statistical metric against at least one predefined security threshold selected from the group consisting of static thresholds, rolling averages, deviation-based thresholds, and percentile-based thresholds.

16. The method of claim 15, further comprising generating a security alert in response to the at least one statistical metric exceeding the at least one predefined security threshold.

17. The method of claim 10, further comprising suppressing duplicate security alerts triggered by the same statistical metric within the sliding window, wherein a suppression state resets upon expiration of the sliding window.

18. The method of claim 10, further comprising correlating multiple types of security events associated with a common security entity within the sliding window, the multiple types selected from the group consisting of authentication events followed by privilege escalations, process executions followed by outbound connections, and repeated failed login attempts followed by successful logins.

19. A distributed system for deterministic sliding-window statistical aggregation over streaming data, configured to execute the method of claim 1, the system comprising OF :an ingestion module configured to receive the streaming data; an aggregation engine configured to perform aggregations using aggregation keys;a shard router configured to distribute the aggregation keys across cache nodes using consistent hashing; a tenant isolation manager configured to enforce data separation using tenant identifiers and namespace separation; anda distributed remote in-memory cache cluster comprising the cache nodes.

20. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 1.

Images