Method and system for filtering events in a data stream

The method and system optimize event filtering in data streams by using tree-structured filters with caching, addressing computational challenges in high-rate event processing and reducing system load.

WO2026238211A1PCT designated stage Publication Date: 2026-11-19F6 JSC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/EA2025/050035
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-16
Filing Date
2025-12-09
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing systems face computational challenges in filtering events with nested structures in data streams, particularly when events arrive at high rates, leading to increased load on computing systems.

Method used

A method and system utilizing tree-structured filters with caching elements of the event's program tree, involving lexical and syntactic analysis, abstract syntax tree construction, and path-based or value-based caching to optimize the filtering process.

Benefits of technology

Reduces the computational load on the system by ensuring efficient and accelerated matching of incoming events with their assigned filters, achieving linear complexity in filtering operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000025_0000
    Figure 00000025_0000
  • Figure 00000026_0000
    Figure 00000026_0000
  • Figure 00000027_0000
    Figure 00000027_0000
Patent Text Reader

Abstract

The present invention relates to the field of information technology, and more particularly to managing events. According to a method for filtering events, first a filter is created in a data stream. In the filter creation step, a filter expression is created which contains a filter condition. A program tree is constructed for the filter expression. When an event comprising at least one field is received, a program tree is constructed for the received event. The program tree constructed for the received event is then traversed. During traversal, elements of the program tree constructed for the received event are cached and, using the cached elements, fields from the event program tree are compared with filter conditions. This comparison is used to decide whether the event complies with the filter expression. Using the present method reduces the load on a computing system when filtering events in a data stream.
Need to check novelty before this filing date? Find Prior Art

Description

H04L 41 / 0604 G06F 18 / 10 METHOD AND SYSTEM FOR FILTERING EVENTS IN A DATA STREAM Field of technology

[0001] The present invention relates to the field of information technology, namely to event management. State of the art

[0002] Filtering incoming data is necessary in many applications. This includes filtering messages and events in various monitoring systems, filtering malicious queries directed at a computer system, and so on. Query languages ​​such as SQL, XPath, and domain-specific languages ​​are widely used for this purpose, allowing for the formulation of filtering and search conditions in log streams, telemetry, and other event sources. In some solutions, filters are defined as text expressions and converted into executable form, which are then applied to incoming data. In existing systems, filtering is typically focused on simple flat data structures and sometimes even requires manual mapping of filters to event fields.

[0003] Applying existing systems to events with a nested structure is significantly complicated by the computationally intensive nature of navigating them. In most applications, events can arrive at rates of up to 100,000 per minute. Therefore, it is necessary to develop technologies that reduce the computational load during filtering and accelerate the entire process of matching incoming events with their assigned filters.

[0004] There are various sources known from the prior art that describe methods for filtering data using program trees.

[0005] In particular, systems and methods for processing filters using hierarchical data and data structures are known, as disclosed in Patent No. US 7,277,885 B2 (published: 02.10.2007; IPC: G06F17 / 30; G06F7 / 00; G06F15 / 16). At least one of the implementations described herein relates to maintaining filters in accordance with hierarchical data associated with the filters. The filters are maintained in the form of a tree structure, which is used to match input data with the filters. The filter hierarchy is an in-memory tree of row segments, where each node corresponds to a row segment and references zero or more filters. When input data is received, the path associated with the input data is split into row segments, and the tree is traversed according to the row segments to find nodes that match the input data (primary matching).The input data is then compared to the filters referenced by the corresponding nodes to find filters that match the input data (secondary matching). As a result, the input data is tested against significantly fewer filters. Furthermore, the primary matching process returns filters in sorted order, facilitating the secondary matching process.

[0006] An optimized query tree is also known, disclosed in US Patent No. 6,253,195 B1 (published: June 26, 2001; IPC: G06F17 / 30). The patent describes abstract syntax trees as a type of filter tree representing procedures for filtering data or other information using logical operations specified, for example, in query-based definitions.

[0007] However, the previously mentioned patents do not describe the use of caching in the context of filtering using program trees. However, there are references that describe caching methods for optimizing data search and processing. In particular, a caching system for optimizing path search is disclosed in US Patent No. 7,283,991 B1 (published October 16, 2007; IPC: G06F7 / 00; G06F9 / 44). This patent describes a caching system designed to optimize class searches in software environments.

[0008] However, none of the prior art sources disclose the use of caching in the context of filtering. The essence of the invention

[0009] The objective of the present invention is to develop a system and method for filtering events in a data stream, ensuring a reduction in the load on a computing system.

[0010] The technical result achieved by this invention is to reduce the load on the computing system when filtering events in a data stream. This technical result is achieved, among other things, through: - use of tree-structured filters; - caching elements of the event's program tree. [UN] This technical result is achieved by filtering events in a data stream. According to this method, a filter is first created. During the filter creation stage, a filter expression is created containing the filter condition. A program tree for this filter expression is constructed. Upon receiving an event that includes at least one field, a program tree for the received event is constructed. The constructed program tree for the received event is then traversed, with the elements of the constructed program tree for the received event being cached during the traversal and the fields from the event's program tree being compared with the filter conditions using the cached elements. Based on this comparison, a decision is made on whether the event matches the filter expression.

[0012] Before constructing the filter expression's program tree, lexical and syntactic analysis of the filter expression can be performed. Then, an abstract syntax tree of the filter expression can be constructed.

[0013] When traversing the program tree of a received event, they can perform a single traversal of the specified tree, during which they can cache paths to fields in the received event.

[0014] When constructing the program tree of a filter expression or event, a tree can be constructed whose nodes are comparators and / or logical operands. If, when comparing the program tree of the filter expression and the program tree of the received event, a logical operand node of the program tree of the filter expression is found, then traversal can continue, checking the child nodes of the program tree of the filter expression, until a comparator node is found. If, when comparing the program tree of the filter expression and the program tree of the received event, a comparator node of the program tree of the filter expression is found, then the value of the found field can be compared with the filter condition contained in the comparator node of the program tree of the filter expression.

[0015] When creating a filter expression, a key can be specified in the comparator. Then, the field in the resulting expression's program tree can be found as follows. The key is found in the comparator, and then, using the specified key and cached paths to fields in the received event, the corresponding field in the received event's program tree can be found using Protobuf reflection.

[0016] They can check the received event until the first field that matches the filter condition, or until the first field that does not match the filter condition.

[0017] After constructing the filter expression's program tree, the tree can be transformed into a linear sequence of blocks, each block containing a comparator and bytecode containing a transition instruction between blocks. Moreover, when comparing the fields of the received event with the filter expression comparators, the compiled bytecode can be used to transition between blocks.

[0018] Additionally, they can accumulate data on the number of matches or mismatches with each comparator. Moreover, when comparing elements of the received event's program tree with comparators of the filter expression's program tree, they can change the sequence of nodes in the filter expression's program tree from those with the highest number of matches or mismatches to those with the fewest.

[0019] When traversing the program tree of a received event, the values ​​of the fields of the specified tree can be additionally cached.

[0020] The technical result is also achieved by a data stream event filtering system in accordance with the present invention. The system includes a filter expression creation module configured to create a filter expression, as well as an event receiving module configured to receive events. Connected to these modules is a conversion unit that receives filter expressions from the filter expression creation module and events from the event receiving module. The conversion unit includes a program tree conversion module configured to convert expressions and events into program trees. The system also includes a tree traversal unit configured to traverse the trees received from the conversion unit. This unit includes a caching module configured to cache the program tree elements of the received event.Connected to the tree traversal unit is a comparison module, configured to compare elements of the received event's program tree with the filtering conditions of the filter expression's program tree during the traversal of the received event's program tree. Furthermore, the comparison module is configured to retrieve cached elements of the received event's program tree from the caching module during the comparison process. The filtering module, which is also an element of the event filtering system in the data stream, is configured to receive comparison results from the comparison module and filter events if the event elements match the filtering conditions.

[0021] The transformation unit may further include a lexer, a parser, and an abstract syntax tree transformation module. The lexer may be configured to perform lexical analysis. The parser may be configured to perform syntactic analysis. The abstract syntax tree transformation module, in turn, may be configured to obtain the results of lexical and syntactic analysis and transform the filter expression into an abstract syntax tree. The program tree transformation module may be configured to obtain an abstract syntax tree and transform it into a program tree.

[0022] The caching module can be configured to cache paths to fields in a received event during a single traversal of the specified tree.

[0023] The program tree conversion module can be configured to construct trees whose nodes are comparators and / or logical operands. The traversal unit can be configured such that upon finding a logical operand node in the filter expression program tree, the traversal continues, checking the child nodes of the filter expression program tree until a comparator node is found. Furthermore, the traversal unit can be configured such that upon finding a comparator node in the filter expression program tree, the value of the found field is compared with the filter condition contained in the comparator node in the filter expression program tree.

[0024] The filter expression program tree comparators may include a key, and the tree traversal unit may be configured to find the comparator key and find the field corresponding to the key in the received event program tree using cached paths to the field in the received event using the specified key and protobuf reflection.

[0025] The comparison module may be configured to check the elements of the received event's program tree up to the first field that matches the filtering condition, or up to the first field that does not match the filtering condition.

[0026] The conversion unit may include a linear sequence conversion module and a bytecode compilation module. The linear sequence conversion module is configured to convert program trees into linear sequences of blocks containing comparators and bytecode compiled by the bytecode compilation module and containing data on the transition logic between blocks of the linear sequence. The bypass block may be configured to use bytecode during bypass to compare the received event with the filter expression comparators.

[0027] The comparison module may be configured with the ability to accumulate data on the number of matches or mismatches with each of the comparators and transmit this data to the program tree conversion module, and the program tree conversion module may be configured with the ability to change the sequence of nodes of the program tree of the filter expression from the largest in terms of the number of matches or mismatches to the smallest.

[0028] The caching module may be configured to cache the values ​​of the fields of the program tree of a received event when traversing the specified tree. Description of drawings

[0029] The subject matter of the present invention is described point by point and clearly expressed in the claims. The above-mentioned objectives, features, and advantages of the invention are apparent from the following detailed description, taken in conjunction with the accompanying drawings, which show:

[0030] Fig. 1 shows a schematic view of the program tree of a filter expression in a first example of implementation in accordance with the present invention.

[0031] Fig. 2 shows a schematic view of the program tree of the received event in the first example of implementation in accordance with the present invention.

[0032] Fig. 3 shows a schematic view of the program tree of the received event in the first example of implementation with indexing in accordance with the present invention.

[0033] Fig. 4 is a schematic view of the program tree of the filter expression in the first example of implementation with permuted nodes in accordance with the present invention.

[0034] Fig. 5 is a schematic view of a program tree of a filter expression in a second example of implementation in accordance with the present invention.

[0035] Fig. 6 is a schematic view of a program tree of a filter expression in a third example of implementation in accordance with the present invention.

[0036] Fig. 7 shows a schematic view of the linear sequence of blocks for the filter expression program tree shown in Fig. 6. Detailed description of the invention

[0037] The following detailed description of the invention includes numerous implementation details intended to provide a clear understanding of the present invention. However, one skilled in the art will readily understand how the present invention may be used with or without these implementation details. In other instances, well-known methods, procedures, and components have not been described in detail in order not to unnecessarily obscure the features of the present invention.

[0038] Furthermore, it is clear from the foregoing description that the invention is not limited to the embodiment described. Numerous possible modifications, changes, variations, and substitutions, while preserving the spirit and form of the present invention, are apparent to those skilled in the art.

[0039] For the purposes of the present invention, the term "event" means a unit of information that represents a change in the state of a system, including, but not limited to: (1) requests, such as HTTP requests, RPC calls, SQL queries, and other forms of access to services or databases; (2) messages, such as messages in data exchange systems, message brokers, queues, logs, or event streams; (3) other signals in data processing systems transmitted between components. A "data stream" in turn means a sequence of structured or unstructured events transmitted in real time or in batch mode between components of a computing system. Data streams may include: (1) requests; (2) messages; (3) logs and telemetry; (4) structured events; (5) real-time streams; (6) batch streams, and other similar objects.

[0040] The disclosed method for filtering events in a data stream is implemented as follows. During the filter creation step, a filter expression containing the filter condition is created. A program tree for this filter expression is constructed. Upon receiving an event that includes at least one field, the program tree for the received event is constructed and the elements of the program tree for the received event are cached. The constructed program tree for the filter expression is then traversed, comparing the fields from the program tree for the received event with the filter conditions using the cached elements. Based on this comparison, a decision is made on whether the event matches the filter expression.

[0041] Creating a filter is necessary to define formal conditions for selecting events that meet specific criteria (filter conditions), for example, when analyzing events, tracking activity, or filtering user or automated queries. A filter is formed based on a logical expression, which can include field value comparison conditions, logical operands, function calls, and other constructs. The created filter is used for subsequent quick matching against incoming events.

[0042] It's important to note that the present invention can be used in a variety of ways. For example, to avoid cluttering the airwaves with unnecessary events when a large number of different events occur in a system, and in particular, operators monitoring the system don't need to see all of these events, only key and unusual ones. Alternatively, the present method can be used to extract specific signatures within an event and block malicious requests based on these extracted signatures. Depending on the application of the present method, filters can be created to meet the required filtering objectives.

[0043] In the first case, filters can be created that block operators from displaying alerts to the user interface about unnecessary execution events, unnecessary network events, local connections, some DNS events, unnecessary events related to environment variables, some file events, etc.

[0044] Filtering parameters for unwanted execution events can be defined based on conditions related to the event type, process path, execution command content, and so on. Adding a process path condition to such filters makes it even more difficult to bypass rules by running a binary file with a custom command string. It's important to note that creating and applying such filters requires careful consideration, as in some cases, ignoring execution processes can lead to process tree corruption. However, without filtering such events, the number of events broadcast may be so high that operators will be unable to identify significant ones. The number of events depends on the scale and complexity of a particular computer system. In some cases, the number of events can reach one billion per minute.

[0045] The filtering parameters for unwanted network events can be set to conditions related to the event type, port number, process path, process groups, etc.; for local connections - to the event type, address (in particular, the local IP address), etc.; for some DNS events - to the event type, process path, address (in particular, the local IP address), etc.; for unwanted events related to environment variables - to the event type, process path, environment variable name, etc.; for some file events - to the event type, process path, file directory, etc.

[0046] Depending on the filtering logic used, filter conditions can be specified either with or without the logical exclusion operator (NOT)—by interpreting the filter as a set of values ​​whose matches cause the event to be blocked or not included in the alert. For example, a condition for blocking events with a type of 1 can be written as event_type == 1 or event_type: 1 . This means that events with a type of 1 are filtered out (blocked, not included in alerts, etc.). The same result can be achieved using the NOT expression (event_type == 1) or NOT (event_type: 1) if the filtering logic is structured differently. Thus, the filtering system allows flexibility in both explicitly specifying logical operators and implicitly—by interpreting the filter as a list of exceptions or permissions. The NOT logical operator will be omitted from this description.For simplicity, only ":" will be used as a comparator for equality comparisons in the filter examples. However, it will be obvious to those skilled in the art that in some programming languages ​​with which the present invention may be implemented, the use of ":" may be unacceptable. However, the use of this operator in the examples should not limit the scope of protection of the present invention with respect to possible implementation methods.

[0047] Depending on their purpose, filters can be simple (checking one field for a specific value), composite (with multiple conditions linked by logical operators), or parameterizable (with the ability to substitute values). Each filter will include at least one comparator (specifically, "==" or ":" from the examples above, as well as comparators greater than ">," less than "<," not equal to "!=", and others). The examples of filter expressions shown in the previous paragraph are examples of simple filter expressions containing only one condition. However, filters can have a more complex structure and contain multiple logical operators at once. For example, if you need to set filter conditions based on both event type and process path, you can define the following filter expression with the logical AND operator: event_type: 1 AND process_path: "a / b / c." In this case, events whose type is 1 and whose process path is a / b / c will be filtered.If you need to create a filter expression that will filter out an event if its type is 1 or the process path is a / b / c, you can use the logical OR operator: event type : 1 OR process_path : “a / b / c”. You can also create a filter expression in which an event will be filtered if its type is 1 or 2 and the process path is a / b / c, using the logical AND and OR operators: (event_type : 1 OR event_type : 2) AND process_path : “a / b / c” . In this case, you can also add exceptions within the filter expression using the logical NOT operator. For example, a filter created based on the filter expression event_type : 1 AND NOT process_path : “a / b / c” will filter out all events of type 1 except those whose process path is a / b / c. All filter expressions described in this paragraph are compound.It's important to note that, based on the above, it's obvious to those skilled in the art how to create filter expressions of any complexity. Therefore, possible implementations of filters in accordance with the present invention are not limited to the examples provided.

[0048] Once a filter expression is created, according to the present invention, a program tree for this filter expression is then constructed. The nodes of this tree are comparators and / or logical operands, depending on the structure of the initial filter expression. Fig. 1 shows an example of a program tree constructed based on the filter expression Ai > 1 AND A.str : "abc".

[0049] Additionally, before constructing the filter expression's program tree, lexical and syntactic analysis can be performed, resulting in the construction of an abstract syntax tree (AST). Transforming a filter expression into an abstract syntax tree allows for the separation of the expression's parsing from its subsequent execution. This ensures the filter's construction logic is independent of the specific language it is described in and simplifies support for various filter formats. Based on the AST, a program tree suitable for direct filter execution can be constructed. This approach ensures the extensibility of the filter system, simplifies syntax error handling, and enables the use of a single execution engine across various programming languages.

[0050] The following comparators can be used for numbers in an abstract syntax tree: LESS - less than, LESS EQUAL - less than or equal, GREATER - greater, GREATER EQUAL - greater than or equal, EQUAL - equal. For strings, the following can be used: EQUAL, REGEX, WILDCARD, and others.

[0051] When an event to be inspected is received, its program tree is also constructed. Once the program tree is constructed, it is cached during the first traversal. Caching can be either path-based or value-based. The specific caching method depends on the structure of the received events to be filtered. Since filters are created in advance and can be reduced to a fixed tree structure, path-based caching is only possible when the structure of the input events is also fixed and repeatable. This is typical, for example, for strongly typed serialization formats such as Protocol Buffers, FlatBuffers, or Apache Avro, where each field has a fixed identifier and location in the tree. In such cases, tree traversal routes can be predefined and stored for reuse.In the case of using less structured formats, such as JSON, YAML, XML or arbitrary key-value structures, where the order of elements can vary or a strict construction scheme may be absent altogether, caching by value is used.

[0052] By-value caching involves storing the variable names from the filter expression tree as keys, and the field values ​​of these variables in the received event as values. Specifically, for the tree shown in Fig. 2, Ai -> 1 and A.str -> "abc" would be cached.

[0053] In cases where it is possible to effectively use path caching, the present invention prefers to use path caching. This is caching by the position of the field in the tree of the received event. During tree traversal, each node is assigned a number (index), which is the ordinal number of the node during the traversal, as shown in Fig. 3. Indexes can also be assigned in advance, for example, when receiving a protobuf. Thus, for example, for the event shown in Fig. 2, the following will be cached: Ai -> [1]; A.str -> [2]. A filter for such an event can be written using keys (paths to a field in the received event) in a comparator. Thus, the filter shown in Fig. 1 can be rewritten as follows: Ai > 1 AND A.str: “abc”.

[0054] Then the comparison will be performed as follows. If a logical operator is found during tree traversal (in the filter example above, the AND operator is found first), the child nodes are checked (Ai > 1 and A.str: "abc"). If a comparator is found, a key is found in it. This key is used to find the field in the received event using cached paths for the field in the received event. The value from this field is compared with the filter condition from the comparator, and the next branch is moved.

[0055] In the current example, during the traversal process, an event is first received and a program tree of the received event is built on its basis, as shown in Fig. 2. This tree is traversed and cached along the paths to its fields: Ai -> [1]; A.str -> [2]. Then, the filter shown in Fig. 1 is applied to the received event. The filter is traversed and the logical operator AND is found. Then, the first comparator is found - Ai > 1 (Ai is greater than 1). A key is found in it, namely “Ai”. Using this key, it is found in the cache that in the received event, a field must be searched for along the path [1]. The value along the path [1] is read and The value 1 is obtained from the field. This value is compared with the comparator filter condition, according to which Ai > 1. Next, the comparator node A. str : “abc” is found (A. str is equal to (EQUAL) “abc”). In it, the key “A. str” is found, by which, in a similar way, it is found in the cache that in the received event it is necessary to search for the field along the path [2]. The value along the path [2] is read and the value “abc” is obtained from the field. This value is compared with the comparator filter condition, according to which A. str: “abc”. Thus, the cache does not store values, but stores the path to the fields. As a result, the specified filter can be applied to all received events that have a prototype matching the prototype of the filter expression tree shown in Fig. 1.

[0056] As mentioned previously, programming languages ​​with a strongly typed serialization format are preferred for implementing path-based caching within the present invention. Specifically, KQL can be used to create filters, and these filters can be applied to Protobuf messages. Each comparator, expressed in KQL, has a key, i.e., the path to a Protobuf field in the message, as well as a value (a string or a number). To find the desired field in the message, this key can be used to find the specified field using Protobuf reflection. The comparison is then performed as described above.

[0057] For example, the message shown in Fig. 2 can be filtered using the filter shown in Fig. 1. This message is a simple (flat) message without nesting. The keys of the filter shown in Fig. 1 are “Ai” and “A.str”. Using reflection (e.g., google.protobuf.reflection in Python, reflection in C++, Message.getField() in Java), the field handle named i (or str) is found and the value is retrieved from this field.

[0058] However, messages may contain nesting. For example, if a message is written as follows: message B { int bi = 5; string str = “abc”;} message A { int ai = 1; B b = 2}, then field b, which is an object of type B, contains its own fields inside. The path cache for this message will be written as follows: Abbi -> [2, 1]; Abstr -> [2, 2]; A.ai -> [1]; Ab -> [2]. In this case, the filter must take such nesting into account. For the specified message type, the filter might look like this: Abbi > 1 AND Abstr: “abc”. The keys of this filter are “Abbi” and “Abstr”. To obtain a value for these keys using reflection, the key string is split as follows: “Abbi” -> [“A”, “b”, “bi”]. Then the fields are obtained recursively. Similarly, fields from a message with any nesting depth can be found.

[0059] It's important to note that when using Protobuf reflection, it's preferable to cache the filter expression's program tree by path rather than by value. This is because reflection-based lookups are extremely computationally expensive.

[0060] In the absence of path caching, a comparison using protobuf reflection to search for fields would be performed as follows. First, the logical AND operator is found. Its child node, which is a comparator, is found: Abbi > 1. Then, the tree of the received event is traversed. During traversal, A is found first. A check is made whether “A” is equal to “Abbi” (“A” == “Abbi”). It is determined that it is not equal and continues traversal, simultaneously checking “A.ai” == “Abbi” -> “Ab” == “Abbi” -> “Abbi” == “Abbi”. The result of the last comparison is True and the value of the field along the path Abbi is read. Abbi = 5 is obtained and it is checked that 5 > 1. After this, it moves to the next child node of the logical AND operator and finds the Abstr comparator: “abc”. Then, similarly, they go through the received message with checks in the following order: “A” == “Abbi” -> “A.ai” == “Abbi” -> “Ab” == “Abbi” -> “Abbi” == “Abbi” -> “Abstr” == “Abstr”. Then, the value of the field along path A is read.b.str, get Abstr = “abc” and check that “abc” == “abc”.

[0061] In the case of using path caching, the logical AND operator is also found first. Its child node, which is a comparator, is found: Abbi > 1. Then the cache is read for “Abbi”, obtaining the path [2, 1]. The value is read along the path [2, 1] in the received event. It obtains Abbi = 5 and checks that 5 > 1. After this, it moves to the next child node of the logical AND operator and finds the comparator Abstr: “abc”. Then, in a similar manner, the value “Abstr” is taken from the cache and the path [2, 2] is obtained. Iterate along this path in the received message and reads the field value. It obtains Abstr = “abc” and checks that “abc” == “abc”.

[0062] If path caching is not used, both trees must be traversed. The filter expression tree must be traversed once, and then, for each comparator node, the received event tree must be traversed twice (that is, for the filter expression tree shown in Fig. 1, the received event tree must be traversed twice). Thus, the complexity of such filtering is quadratic: O(n*m) -> O(N) 2 ), where n is the number of nodes in the filter expression tree, and am is the number of nodes in the received event tree. With path caching, the filter expression tree is traversed only once, and the fields of the received event tree are accessed via pre-found and cached paths, making the complexity (O(n*l) -> O(N)) linear. This approach significantly accelerates the filtering process and reduces the load on the computing system.

[0063] Received events can be checked in batches. In this case, each tree of the received event is checked in turn. This check can be performed either until the first tree that matches the filtering conditions or until the first one that does not, depending on the settings. That is, for "positive filters," an event is passed when all listed trees of conditions are met. For "negative filters," an event is rejected when at least one condition is met.

[0064] Additionally, they can accumulate data on the number of matches or mismatches with each filter condition (in comparators). Moreover, when comparing elements of the received event's program tree with comparators of the filter expression's program tree, they can change the sequence of nodes in the filter expression's program tree from those with the highest number of matches or mismatches to those with the fewest. This is especially useful when filtering with negative filters, where at least one condition must be met to reject an event. This eliminates the need to waste computational resources on checking all filter conditions for each received event.

[0065] For example, if we receive three different events, the first of which is A{ i = 0; str = “abc”}, the second is A{ i = 2; str = “abc”}, and the third is A{ i = 0; str = “xyz”}, and we apply the filter shown in Fig. 1 to each of them, then the number of matches with the filtering condition Ai > 1 will be equal to 1, and the number of matches with the filtering condition A. str : “abc” will be equal to 2. Thus, the second filtering condition is triggered more often than the first. Then the program tree of the filter expression shown in Fig. 1 can be rebuilt as shown in Fig. 4. Now, when checking events using the specified filter, the condition A. str : “abc” will be checked first, since it is triggered more often. This will additionally reduce the load on the computing system.

[0066] When traversing the program tree of a received event, the values ​​of the fields of the specified tree can be cached. This is useful in cases where a filter contains two filtering conditions on the same field. For example, the filter (Ai > 1 AND (A. str: “abc” OR A. str: “xyz”)), shown in Fig. 5, checks the field A. str twice. Then, when traversing the tree of the received event shown in Fig. 2, the values ​​of the fields Ai and A. str will be cached. When comparing with the filtering conditions of the program tree of the filter shown in Fig. 5, there will be no need to search for the path to the field A. str a second time, since the value of this field in the received message can be simply taken from the cache. This also helps reduce the load on the computing system during the filtering process.

[0067] In this case, traversal and comparison will be performed as follows. First, as mentioned earlier, when traversing the tree of the received event, the paths to its fields are cached, writing to the cache: Ai -> [1]; A. str -> [2]. Then, a comparison with the filter is started. To do this, the tree of the filter expression is traversed and first the logical operator AND is found. Its child node, which is the comparator, is found: Ai > 1. Then, the cache is read for “Ai”, path [1] is obtained. The value along path [1] in the received event is read. They get Ai = 1, write “Ai = 1” to the cache and check that 1 > 1. After this, they move to the next child node of the logical operator AND and find the logical operator OR. They look for its child node and find the comparator A. str: “abc”. Then, in a similar way, they take the value of “A. str” from the cache and path [2] is obtained. They traverse this path in the received message and read the value of the field. They get A. str = “abc” and check, write “A. str = “abc”” to the cache and that “abc” == “abc”.Then they find the second child node of the logical OR operator: A.str: "xyz." In this case, they won't search the cache again for the path to the A.str field in the received event, but will instead take the value of this field from the cache directly, namely A.str = "abc." Then they'll perform the comparison "xyz" == "abc."

[0068] The following optimization is also possible. After constructing the filter expression program tree, it can be transformed into a linear sequence of filter conditions—a filter conditions array. The program tree has a deterministic order in which nodes should be traversed when checking whether an event matches the filter conditions specified by the expression. During such a traversal, it is necessary to first descend to the leftmost node of the tree (or to the topmost node, as applied to the trees in Figures 1, 3, and 4), perform the calculations, then go via the parent node to the right-hand neighboring node (the lower-hand neighbor, as applied to the trees in Figures 1, 3, and 4), if one exists, and repeat the descent to the rightmost node (the lower-hand node, as applied to the trees in Figures 1, 3, and 4) until the tree is completely traversed. However, when evaluating logical expressions, it is possible to skip nodes that do not affect the subsequent result.This can be achieved by transforming the program tree into a linear sequence of filter conditions.

[0069] For example, if for the filter (A:1 AND B:1) OR C:1 we receive the event A = 0, B = 1, C = 1, then when calculating A:1 AND B:1, it is immediately clear that, since A = 0, checking the value of B is pointless, as this will not affect the truth of the entire expression. Moreover, returning one level higher to the logical operator AND and then descending to the comparator B:1 creates additional unnecessary load on the computing system. Therefore, it is possible to immediately proceed to checking C:1, reducing the load during calculation and increasing the overall calculation speed.

[0070] To achieve this, the entire nested tree structure can be transformed, while preserving order, into a linear sequence (array) of blocks (filter conditions), each of which inherits the original function of the corresponding comparator node in the original tree (in traversal order). It is also possible to add a hint to each block, indicating which block to proceed to next (skipping one or more subsequent ones) to continue evaluating the logical expression without checking nodes that do not affect the overall result. To achieve this, each block of the linear sequence includes a filter condition and bytecode containing the logic for transitioning between nodes.

[0071] Fig. 6 shows the program tree for the filter ((NOT name : “abc” AND label : “allowed”) AND (i : 1 OR hint : “true”)) OR state : “false”. In the case of the specified filter, if an event is received in which name = “abc”, there is no point in checking the conditions label : “allowed”, i : 1 and hint : “true”, but it is possible to immediately proceed to checking the condition state : “false”. Otherwise, it is necessary to go directly to the next block, namely label : “allowed”.

[0072] Also, if the remaining checks yield negative results. Thus, the tree shown in Fig. 6 can be represented as a linear sequence of blocks containing filter conditions and bytecodes containing the logic for transitions between nodes, as shown in Fig. 7. It is important to note that the linear sequence according to the present invention does not include the END block; however, for ease of understanding, it has been added to Fig. 7 to indicate the completion of the comparison process. Thus, the described possible optimization consists of avoiding visits to unnecessary nodes of the filter expression tree, i.e., those nodes that will not be evaluated anyway due to the obtained intermediate result.

[0073] Bytecodes with jump instructions can be written similar to conditional jumps in assembly languages ​​(je - jump if equals, jne - jump if not equals, etc.), as shown in Figure 7. Blocks with logical conditions are evaluated in left-to-right order. If the condition of the current block is satisfied, the evaluation proceeds to the immediately following block with a true result. Otherwise, a hint instruction is used to jump to the block at the index specified in the hint (vector indices assigned within path caching can be used). The true / false from the hint is also used as the current result. This further reduces the load on the computing system and speeds up the verification process.

[0074] Thus, the described method allows for effective event filtering. Efficiency in this case refers to increased processing speed and reduced load on the computing system.

[0075] To implement the described method, a data stream event filtering system may be used in accordance with the present invention. The system includes a filter expression creation module configured to create a filter expression, as well as an event receiving module configured to receive events. Connected to said modules is a conversion unit that receives filter expressions from the filter expression creation module and events from the event receiving module. The conversion unit includes a program tree conversion module configured to convert expressions and events into program trees. The system also includes a tree traversal unit configured to traverse the trees received from the conversion unit. Said unit includes a caching module configured to cache the program tree elements of the received event.Connected to the tree traversal unit is a comparison module, configured to compare elements of the received event's program tree with the filtering conditions of the filter expression's program tree during the traversal of the received event's program tree. Furthermore, the comparison module is configured to retrieve cached elements of the received event's program tree from the caching module during the comparison process. The filtering module, which is also an element of the event filtering system in the data stream, is configured to receive comparison results from the comparison module and filter events if the event elements match the filtering conditions.

[0076] The terms "module", "block" and similar terms used in the description of the technical solution are used to designate computer entities that may be hardware / equipment (e.g., a device, instrument, apparatus, apparatus, component part of a device, such as a processor, microprocessor, integrated circuit, printed circuit board, including an electronic printed circuit board, breadboard, motherboard, etc., a microcomputer, etc.), software (e.g., executable program code, a compiled application, a service, a program module, a part of software or program code, etc.) and / or microprogram (in particular, firmware).For example, a component may be a process running on a processor (a processor), an object, executable code, program code, a file, a program / application, a function, a method, a (software) library, a subroutine, a coroutine, and / or a computing device (e.g., a microcomputer or a computer), or a combination of software or hardware components. Thus, in a particular case, an application running on a server may be a component / module, and the server, in turn, may be a component / module. It is worth noting that at least one component / module may be part of a process. A component / module may be located on a single computing device (e.g., a microcomputer, a microprocessor, a printed circuit board, etc.) and / or may be distributed / partitioned among several computing devices.

[0077] The filter expression creation module can be implemented as a program that allows filter expressions to be written in any known programming language. Similarly, all modules and blocks described above and below can be implemented using known programming language tools as individual functions (either already contained in the libraries being used or custom-written). It is important that the blocks and modules perform the functions described above and below.

[0078] The transformation unit may further include a lexer, a parser, and an abstract syntax tree transformation module. The lexer may be configured to perform lexical analysis. The parser may be configured to perform syntactic analysis. The abstract syntax tree transformation module, in turn, may be configured to obtain the results of lexical and syntactic analysis and transform the filter expression into an abstract syntax tree. The program tree transformation module may be configured to obtain an abstract syntax tree and transform it into a program tree.

[0079] The caching module may be configured to cache elements of the program path tree to fields in a received event, assigning vector indices to the fields during a single traversal of the specified tree.

[0080] The program tree conversion module can be configured to construct trees whose nodes are comparators and / or logical operands. The traversal unit, in turn, can be configured such that upon finding a logical operand node in the filter expression program tree, the traversal continues, checking the child nodes of the filter expression program tree until a comparator node is found. Furthermore, the traversal unit can be configured such that upon finding a comparator node in the filter expression program tree, the value of the found field is compared with the filter condition contained in the comparator node in the filter expression program tree.

[0081] The filter expression program tree comparators may include a key, and the tree traversal unit may be configured to find the comparator key and find the field corresponding to the key in the received event program tree using cached paths to the field in the received event using the specified key and protobuf reflection.

[0082] The comparison module may be configured to check the elements of the received event's program tree up to the first field that matches the filtering condition, or up to the first field that does not match the filtering condition.

[0083] The conversion unit may include a linear sequence conversion module and a bytecode compilation module. The linear sequence conversion module is configured to convert program trees into linear sequences of blocks containing filter conditions (comparators) and bytecode compiled by the bytecode compilation module and containing data on the transition logic between blocks of the linear sequence. The traversal unit may be configured to use bytecode during traversal, comparing the received event with the conditions (comparators) of the filter expression.

[0084] The comparison module can be configured with the ability to accumulate data on the number of matches or mismatches with each of the filtering conditions (comparators) and transmit this data to the program tree conversion module, and the program tree conversion module can be configured with the ability to change the sequence of nodes of the program tree of the filtering expression from the largest in terms of the number of matches or mismatches to the smallest.

[0085] The caching module may be configured to cache the values ​​of the fields of the program tree of a received event when traversing the specified tree.

[0086] It is important to note that any additional elements and functions of the data stream event filtering system described above may be used individually, simultaneously, or in any combination. Implementing the system with any additional element will lead to the achievement of additional technical results described in the present invention, in addition to the primary technical result. Furthermore, any of the additional features of the system may be interpreted as an additional feature of the data stream event filtering method. Similarly, any of the additional features of the data stream event filtering method may be interpreted as an additional feature of the data stream event filtering system in accordance with the present invention.

[0087] These materials present a preferred disclosure of the implementation of the claimed technical solution, which should not be used as limiting other, particular embodiments of its implementation that do not go beyond the scope of the requested scope of legal protection and are obvious to specialists in the relevant field of technology.

Claims

1. Invention formula 1. A method for filtering events in a data stream by which: at the filter creation stage: - create a filter expression containing the filter conditions; - construct a program tree of the filter expression; when receiving an event that includes at least one field: - construct a software tree of the received event; - they traverse the constructed software tree of the received event, and during the traversal they cache the elements of the constructed software tree of the received event and compare the fields from the software tree of the event with the filtering conditions, using the cached elements; - based on the comparison, a decision is made about whether the event matches the filter expression.

2. Method according to I. 1, characterized in that before constructing the program tree of the filter expression: - conduct lexical and syntactic analysis; - construct an abstract syntax tree.

3. The method according to item 1 or item 2, characterized in that when traversing the program tree of the received event, a single traversal of the specified tree is performed, during which the paths to the fields in the received event are cached.

4. The method according to item 3, characterized in that when constructing the program tree of the filtering expression or event, a tree is constructed whose nodes are comparators and / or logical operands, and if, when comparing the program tree of the filtering expression and the program tree of the received event, a logical operand node of the program tree of the filtering expression is found, then the traversal continues, checking the child nodes of the program tree of the filtering expression, until a comparator node is found.

5. The method according to item 3 or item 4, characterized in that when constructing the program tree of the received event, indexes are assigned to the fields, and if, when comparing the program tree of the filtering expression and the program tree of the received event, a comparator node of the program tree of the filtering expression is found, then the value of the found field is compared with the filtering condition contained in the comparator node of the program tree of the filtering expression.

6. The method according to item 5, characterized in that when creating the filtering expression, a key is specified in the comparator, and the field in the program tree of the received event is found as follows: - find the key contained in the comparator of the program tree of the filter expression; - find the field corresponding to the key in the program tree of the received event by the specified key and cached paths to the fields in the received event using protobuf reflection.

7. The method according to paragraph 5, characterized in that the received event is checked up to the first field that matches the filtering condition, or up to the first field that does not match the filtering condition.

8. The method according to item 1 or item 2, characterized in that after constructing the program tree of the filtering expression, said tree is transformed into a linear sequence of blocks, in which each block includes a comparator and bytecode containing an instruction for transition between blocks, and when comparing the fields of the received event with the comparators of the filtering expression, the compiled bytecode is used for transition between blocks.

9. The method according to any of paragraphs 1-8, characterized in that data on the number of matches or mismatches with each of the comparators is accumulated, and when comparing the elements of the program tree of the received event with the comparators of the program tree of the filtering expression, the sequence of nodes of the program tree of the filtering expression is changed from those with the greatest number of matches or mismatches to those with the least.

10. The method according to any of paragraphs 1-9, characterized in that when traversing the program tree of the received event, the field values ​​of the specified tree are additionally cached.

11. A system for filtering events in a data stream, including: - a module for creating filter expressions, configured with the ability to create a filter expression; - an event receiving module configured with the ability to receive events; a conversion unit configured to receive filter expressions from a filter expression creation module and events from an event receiving module and including a program tree conversion module configured to convert expressions and events into program trees; - a traversal unit configured with the ability to traverse the trees received from the transformation unit and including a caching module configured with the ability to cache the elements of the software tree of the received event, - a comparison module connected to the tree traversal unit and configured with the ability to compare the elements of the software tree of the received event with the filtering conditions from the software tree of the filtering expression during the process of traversing the software tree of the received event, wherein the comparison module is configured with the ability to receive cached elements of the software tree of the received event from the caching module during the comparison process; - a filtering module configured with the ability to receive comparison results from the event comparison and filtering module when event elements match the filtering conditions.

12. The system according to claim 11, characterized in that the conversion unit further includes a lexer configured with the ability to perform lexical analysis, a parser configured with the ability to perform syntactic analysis, and an abstract syntax tree conversion module configured with the ability to obtain the results of lexical and syntactic analysis and convert the filter expression into an abstract syntax tree, wherein the program tree conversion module is configured with the ability to obtain an abstract syntax tree and convert it into a program tree.

13. The system according to paragraph 11 or paragraph 12, characterized in that the caching module is configured with the ability to cache paths to fields in the received event during a single traversal of the specified tree.

14. The system according to claim 13, characterized in that the module for converting into program trees is configured with the ability to construct trees whose nodes are comparators and / or logical operands, and the traversal unit is configured in such a way that when a node of a logical operand of the program tree of the filter expression is found, the traversal continues, checking the child nodes of the program tree of the filter expression, until a comparator node is found.

15. The system according to claim 14, characterized in that the bypass unit is configured in such a way that when a node of the comparator of the program tree of the filtering expression is found, the value of the found field is compared with the filtering condition contained in the node of the comparator of the program tree of the filtering expression.

16. The system according to item 14 or item 15, characterized in that the comparators of the program trees of the filter expression include a key, and the tree traversal unit is configured with the ability to find the comparator key and find the field corresponding to the key in the program tree of the received event along the cached paths to the field in the received event using the specified key and protobuf reflection.

17. The system according to item 15 or item 16, characterized in that the comparison module is configured with the ability to check the elements of the software tree of the received event up to the first field that matches the filtering condition, or up to the first field that does not match the filtering condition.

18. The system according to item 11 or item 12, characterized in that the conversion unit includes a linear sequence conversion module configured with the ability to convert program trees into linear sequences from blocks that include comparators and bytecode compiled by the bytecode compilation module and containing instructions for transitioning between blocks of a linear sequence, and transmitting it to the bypass unit, which is configured with the ability to use bytecode when comparing the received event with the comparators of the filter expression.

19. The system according to any one of paragraphs 11-18, characterized in that the comparison module is configured with the ability to accumulate data on the number of matches or mismatches with each of the comparators and transmit this data to the module for converting into program trees, and the module for converting into program trees is configured with the ability to change the sequence of nodes of the program tree of the filter expression from the largest in terms of the number of matches or mismatches to the smallest.

20. The system according to any of paragraphs 11-19, characterized in that the caching module is configured with the ability to cache the values ​​of the fields of the program tree of the received event when traversing said tree.