34 results about "Data stream management system" patented technology

A “Query Optimizer” provides a cost estimation metric referred to as “Maximum Accumulated Overload” (MAO). MAO is approximately equivalent to maximum system latency in a data stream management system (DSMS). Consequently, MAO is directly relevant for use in optimizing latencies in real-time streaming applications running multiple continuous queries (CQs) over high data-rate event sources. In various embodiments, the Query Optimizer computes MAO given knowledge of original operator statistics, including “operator selectivity” and “cycles / event” in combination with an expected event arrival workload. Beyond use in query optimization to minimize worst-case latency, MAO is useful for addressing problems including admission control, system provisioning, user latency reporting, operator placements (in a multi-node environment), etc. In addition, MAO, as a surrogate for worst-case latency, is generally applicable beyond streaming systems, to any queue-based workflow system with control over the scheduling strategy.

A computer is programmed to accept a command to create a new aggregation defined by a user during execution of continuous queries on streams of data. The computer is further programmed to thereafter accept and process new continuous queries using the new aggregation, in a manner similar to built-in aggregations. The user typically writes a set of instructions to perform the new aggregation, and identifies in the command, a location of the set of instructions. In response to such a command, the computer creates metadata identifying the new aggregation. The metadata is used to instantiate one aggregation for each group of data in a current window, grouped by an attribute identified in a new query.

Facilitating flexible windows in querying continuous data streams. In one embodiment, a software code is associated with a window, wherein execution of the software code determines the values to be included in the window. On receiving a query indicating the window, the result of the query is generated based on the values in the window determined by execution of the software code. The software code may be identified by an identifier and thus used for different queries on the same or different continuous data streams.

Simplifying a user's task in specifying a family of logics, with each logic designed to potentially select different subsets of values from the same continuous data stream at the same time instant. In one embodiment, all the logics are together specified by a common user specified software code design to accept a set of parameters, with different logics resulting from execution of the software code with different parametric values. Thus, when closely related family of logics are to be developed, the common software code may be formulated, with the code being designed to accept parameters and providing different logics for different sets of parametric values corresponding to the parameters.

Simplifying a user's task in specifying a family of logics, with each logic designed to potentially select different subsets of values from the same continuous data stream at the same time instant. In one embodiment, all the logics are together specified by a common user specified software code design to accept a set of parameters, with different logics resulting from execution of the software code with different parametric values. Thus, when closely related family of logics are to be developed, the common software code may be formulated, with the code being designed to accept parameters and providing different logics for different sets of parametric values corresponding to the parameters.

A method and system are disclosed for operating a high speed data stream management system which runs a query plan including a set of queries on a data feed in the form of a stream of tuples. A predicate prefilter is placed outside the query plan upstream of the set of queries, and includes predicates selected from those used by the queries. Predicates are selected for inclusion in the prefilter based on a cost heuristic, and predicates are combined into composites using a rectangle mapping heuristic. The prefilter evaluates the presence of individual and composite predicates in the tuples and returns a bit vector for each tuple with bits representing the presence or absence of predicates in the tuple. A bit signature is assigned to each query to represent the predicates related to that query, and a query is invoked when the tuple bit vector and the query bit signature are compatible.

Data stream query mediation may utilize a query handler configured to receive a query from a stream application to be applied against a stream of data including multiple tuples representing events. A stream window manager may be configured to express the query in a specification which defines a window including a subset of the tuples, the specification defining content of the window as a range of the tuples having a range attribute over which the content is specified, defining when to update the window using a synchronization attribute specifying a movement of the window over time with respect to the content, and defining an evaluation of the content of the window using an evaluation attribute specifying when to perform the evaluation. A query translator may be configured to translate the query including mapping the range attribute, the synchronization attribute, and the evaluation attribute to a stream processing language of a data stream management system (DSMS), to thereby obtain a translated query. A DSMS mediator may be configured to provide the translated query to the DSMS for processing therewith.

A method and system are disclosed for operating a high speed data stream management system which runs a query plan including a set of queries on a data feed in the form of a stream of tuples. A predicate prefilter is placed outside the query plan upstream of the set of queries, and includes predicates selected from those used by the queries. Predicates are selected for inclusion in the prefilter based on a cost heuristic, and predicates are combined into composites using a rectangle mapping heuristic. The prefilter evaluates the presence of individual and composite predicates in the tuples and returns a bit vector for each tuple with bits representing the presence or absence of predicates in the tuple. A bit signature is assigned to each query to represent the predicates related to that query, and a query is invoked when the tuple bit vector and the query bit signature are compatible.

Methods and systems of managing physician payment data (PPD) may involve a data compilation system for compiling physician payment data from multiple sources; a data processing system for processing the compiled physician payment data; a data management system for creating a searchable database from the processed physician payment data; and a data distribution system for providing access to the searchable database. Exemplary embodiments of the invention may include a computer-implemented system for automated data compilation, processing, management, analysis, and distribution, such as facilitating online input, output, access and retrieval of PPD using software accessed via a network portal on the Internet. Numerous other aspects are provided.

A data stream management system for accessing mass data and method thereof is disclosed. The system includes: a client computer and a number of distributed server groups. The client computer and the distributed server groups are connected via network. Each of the distributed server groups including: a determination unit, a dividing unit, a transmitting unit, a number of distributed servers and a dispatching server. The system divides a main data into a number of data sections and stores them in the distributed servers of different distributed server groups. The system can quickly integrate the distributed data sections back into the main data by uses of a global index.

A data flow management system and method in which an application and its clients are made aware of the available credits for each type of transfer before the transfer is attempted. This enables the clients to transmit packets only when the RX side has issued a sufficient number of credits to insure that the transmission will not be stalled. The invention eliminates the need for FIFO buffers in the PCI-Express core, since the application will not transmit packets to the core until the required number of credits for the particular transfer type is available. Therefore, packet transmissions do not require buffering in the core, as they are only sent when they can be sent all the way through the core to the link. The efficient back-to-back transfer support enabled by this protocol increases the overall throughput and efficiency of the system as transfers through the application interface have fewer gaps therebetween, because the credit management protocol with its back-to-back feature reduces the idle times between packets traversing the application interface. An enhanced Quality of Service feature is also introduced by creating an environment where the low-latency packets can receive a highest attention and the lowest transmission wait time.

A computer-implemented method comprises receiving a plurality of data streams into a data stream management system, wherein each data stream is obtained from a data streaming source and a reliance level is associated with each of the data streaming sources. When a continuous query is registered that specifies a reliance level attribute, the query is executed over the plurality of data streams using only those data streams obtained from a data streaming source associated with a suitable reliance level. For example, the reliance level attribute may establish a minimum reliance level, such that data stream having a lesser reliance level are not used. CQL may be enhanced with such an extra attribute which allows end users to specify the desired reliance level when submitting a query.

CAPTCHA (Completely Automated Public Test to tell Computers and Humans Apart) data generation methods for use in an electronic device and related management systems are provided. First, the electronic device determines a first data set according to at least one first data corresponding to an operation to be performed, wherein the first data represents sensitive data corresponding to the operation. Then, the electronic device generates a group of CAPTCHA data corresponding to the first data set according to the first data. The electronic device may be a server or a client. When the electronic device is the client, the client obtains at least one generation module from the server to determine the first data set, and generate the CAPTCHA data. In some embodiments, during a data transmission procedure, the client performs the operation with the server using the CAPTCHA data.

A data flow management system and method in which the application and its clients are made aware of the available credits for each type of transfer before the transfer is attempted. This enables the clients to transmit packets only when the RX side has issued a sufficient number of credits to insure that the transmission will not be stalled. The invention eliminates the need for FIFO buffers in the PCI-Express core, since the application will not transmit packets to the core until the required number of credits for the particular transfer type is available. Therefore, packet transmissions do not require buffering in the core, as they are only sent when they can be sent all the way through the core to the link. The efficient back-to-back transfer support enabled by this protocol increases the overall throughput and efficiency of the system as transfers through the application interface have fewer gaps therebetween, because the credit management protocol with its back-to-back feature reduces the idle times between packets traversing the application interface. An enhanced Quality of Service feature is also introduced by creating an environment where the low-latency packets can receive a highest attention and the lowest transmission wait time.

Disclosed is a rate-adaptive data stream management system (DSMS) which is a precedence task, which has a descendent task, and a plurality of data channels and a plurality of feedback channels, the system including: a feedback analyzer which analyzes a control command transmitted from the descendent task; and a data channel selector which controls transmission of a data stream through the data channel with the descendent task based on the analysis result.

The invention provides a data warehouse system. A basic framework is constructed by relying on the storage and calculation capabilities of the big data platform; a scheduling management system, a bigdata platform batch processing framework module, a development operation and maintenance management system, a big data platform system, a big data batch processing task development tool, a data management system and other systems are developed. The capacity of the data warehouse is expanded, and the aspects of data architecture, data modeling, data management, the data warehouse, data development,operation and maintenance, task scheduling and the like are improved. According to the data warehouse, data storage can be carried out, and data exporting can be carried out on the obtained data based on the configuration information exported by the data, so that data exporting operation is supported. In addition, the data processing requirement can be received through the big data platform batchprocessing framework module, data needed by the data processing requirement are processed according to the data processing requirement, the data processing function is achieved, and the function of the database is expanded.

The invention relates to the technical field of data management systems, and particularly to a main-data system used for realizing enterprise internal-data standardization. The main-data system comprises: a data reading / writing module, which is used for reading / writing data of all service systems; a main-data management module, which comprises a main-data storage sub-module and a main-data new-creating sub-module, wherein the main-data storage sub-module stores a main-data list, the main-data new-creating sub-module is used for adding new main data into the main-data list, and the main data include a data cleaning rule; and a data operation module, which is used for carrying out matching on the data in all the service systems through the data reading / writing module according to the data cleaning rule of the main data, and cleaning and replacing matching data according to the data cleaning rule. The main-data system used for realizing enterprise internal-data standardization provided bythe invention can solve the problem of main-data confusion existing in existing enterprise service systems.

Disclosed is a rate-adaptive data stream management system (DSMS) which is a precedence task, which has a descendent task, and a plurality of data channels and a plurality of feedback channels, the system including: a feedback analyzer which analyzes a control command transmitted from the descendent task; and a data channel selector which controls transmission of a data stream through the data channel with the descendent task based on the analysis result.

Provided are method and device for managing a memory in a data stream management system (DSMS) of a portable device. The method includes moving data of a selected memory region that has a low priority to a secondary storage and storing a received data stream in the selected memory region.

The invention discloses an agent cooperation method based on multi-modal data fusion, and belongs to the technical field of multi-modal data fusion, and the method comprises the steps of dynamically collecting multi-source heterogeneous data, automatically distributing agent roles based on reinforcement learning, and optimizing task execution efficiency. The method comprises the following steps: identifying data, classifying the data according to the characteristics of the data, and establishing an efficient data flow management system for each identified data type to control the speed and sequence of the data flowing into the system from different sources, define the environment of agent operation and create a detailed capability model for each agent; historical data and a simulation scene are used to train an agent, when a new task arrives, the system determines a most suitable agent role allocation scheme in real time by using a trained reinforcement learning algorithm according to a current environment state, and feeds back an actual result to the reinforcement learning module during the task execution period of the agent.

Provided are a method and apparatus for controlling load shedding for excluding data streams input into a data stream management system (DSMS). A method of selecting data not to be in the data process may be determined. A load shedding factor may also be determined. Data streams of the data process may be processed by applying the method of selecting the data according to the load shedding factor.

Provided are a method and apparatus for controlling load shedding for excluding data streams input into a data stream management system (DSMS). A method of selecting data not to be in the data process may be determined. A load shedding factor may also be determined. Data streams of the data process may be processed by applying the method of selecting the data according to the load shedding factor.

An apparatus and a method of optimizing queries through scheduling User Defined Operators (UDOs) in a Data Stream Management System (DSMS), are provided. The apparatus includes a query optimizer configured to receive queries, each of the queries including the UDOs and data streams subject to respective operations of the UDOs, and group the data streams and UDOs into scheduling units, each of the scheduling units including one of the data streams and one of the UDOs. The apparatus further includes a scheduler configured to schedule an execution order in which the scheduling units are executed.

The description relates to cloud-edge topologies. Some aspects relate to cloud-edge applications and resource usage in various cloud-edge topologies. Another aspect of the present cloud-edge topologies can relate to the specification of cloud-edge applications using a temporal language. A further aspect can involve an architecture that runs data stream management systems (DSMSs) engines on the cloud and cloud-edge computers to run query parts.

An apparatus and a method of optimizing queries through scheduling User Defined Operators (UDOs) in a Data Stream Management System (DSMS), are provided. The apparatus includes a query optimizer configured to receive queries, each of the queries including the UDOs and data streams subject to respective operations of the UDOs, and group the data streams and UDOs into scheduling units, each of the scheduling units including one of the data streams and one of the UDOs. The apparatus further includes a scheduler configured to schedule an execution order in which the scheduling units are executed.

A computer-implemented method comprises receiving a plurality of data streams into a data stream management system, wherein each data stream is obtained from a data streaming source and a reliance level is associated with each of the data streaming sources. When a continuous query is registered that specifies a reliance level attribute, the query is executed over the plurality of data streams using only those data streams obtained from a data streaming source associated with a suitable reliance level. For example, the reliance level attribute may establish a minimum reliance level, such that data stream having a lesser reliance level are not used. CQL may be enhanced with such an extra attribute which allows end users to specify the desired reliance level when submitting a query.

Described herein is a compliance dataflow management system. The system may include a plurality of different databases. The system may be configured to provide for workflow that are in conformance of regulations and / or requirements for specific jurisdictions. Furthermore, aspects of the system may allow for automatic updating of compliance systems in a manner that allows for resource savings for automated compliance software. Additionally, the system may include automated auditing systems, allowing for compliance to be determined in a quick and resource efficient manner while allowing for privacy and security to be maintained.