9 results about "Distributed data flow" patented technology

The invention discloses a real-time processing method and system for distributed data streams, and particularly relates to the technical field of distributed data processing, and the method comprises the following steps: collecting multi-source data streams and marking priorities, combining data types and priorities to carry out adaptive segmentation, carrying out load balancing distribution on processing units, and resolving data conflicts; and incremental integration and caching of hotspot data are carried out, and full-link tracing and dynamic tuning are carried out. The system comprises five modules, namely an acquisition labeling module, a segmentation module, a parallel processing module, an integration optimization module and a tracing tuning module. According to the real-time processing method and system for the distributed data flow, priority processing of core data is achieved, data consistency is guaranteed, the resource utilization rate and the system maintainability are improved, and the processing requirements for high real-time performance and high reliability are met.

Techniques for log storage in distributed data streaming systems are described. A cluster of brokers receive log records from publishers and send log records to subscribers. The log is represented as a group of segments, each segment subdivided into chunks. Metadata describes the log structure. Log records are stored in chunks at least in a remote storage location shared amongst the brokers in the cluster.

The invention provides a data stream adaptive scheduling method, system and device and a medium, and belongs to the technical field of distributed data stream processing. The method comprises the following steps: acquiring NiFi system operation index data, and extracting standardized features by using feature engineering; a machine learning model combination comprising a neural network, a gradient elevator and a reinforcement learning model is constructed for prediction analysis, and an execution time prediction value, a failure probability prediction value and scheduling strategy parameters are generated; dynamically adjusting system configuration parameters based on the prediction result; risk assessment and graded alarm are realized through a multi-stage rule engine; performing real-time state display and configuration management by adopting a visual engine; and finally, an optimization suggestion report is generated through the data analysis platform. According to the invention, adaptive scheduling and intelligent operation and maintenance of the data stream processing system are realized, and the resource utilization efficiency and the operation stability of the system are effectively improved.

The invention relates to the technical field of computer architecture, and provides a computer system and a data processing method, and the system comprises the steps that an on-chip network distributes data traffic according to an address interleaving strategy; the data link module comprises a plurality of data link units, and each data link unit comprises a plurality of data links; the network transmission module comprises a plurality of network transmission layers, the plurality of network transmission layers are in communication connection with the network-on-chip, each network transmission layer is in communication connection with the plurality of data links in one data link unit, and the network transmission layers receive first data sent by the network-on-chip and send the first data to the network-on-chip; distributing the first data to at least one data link in the data link unit according to a preset scheduling strategy; and transmitting second data received from at least one data link in the data link unit to the network-on-chip. According to the invention, through a one-to-many hierarchical link architecture, when a certain link fails, the link can be switched to other links, so that the stability and reliability of the system are enhanced.

A modular and distributed architecture for data stream processing and analysis is described to incorporate data stream analytics capabilities, called Data Stream Analytics Service (DSAS) in the IoT / M2M service layer. Each service layer node hosting DSAS can be split into two independent modules, Stream Forwarder and Stream Analytics Engine. Stream Forwarder is a light weight processing modules that can be responsible for data preprocessing and routing. Stream Analytics Engine is responsible for performing actual analytics on the data stream. Separating the two functionalities enables the service layer nodes to efficiently distribute stream analytics tasks across multiple nodes.

The invention provides a product information pushing method, device, medium and system for a mobile banking, and the method comprises the steps: collecting user transaction behavior data of the mobile banking through employing a data processing pipeline tool under the condition of obtaining user authorization, and caching the user transaction behavior data through employing a distributed message caching queue; summarizing the cached user transaction behavior data by adopting a distributed data flow calculation engine according to a preset dimension rule to obtain a summarizing result, and storing the summarizing result into a column database; and querying the data in the column database by adopting the query statement to obtain user behavior data and product transaction information, processing the user behavior data and the product transaction information to obtain potential product information, and pushing the potential product information. Therefore, the problem that in the prior art, a platform can only query the user funnel diagram to achieve a display function, but cannot push the product information for the customer loss rate in the user funnel is solved.

The application provides an interface parallel distribution method and system for comprehensive monitoring system reconstruction, and the system comprises an interface parallel distribution system for connecting a new platform, an old platform and each specialty; the interface parallel distribution system comprises a data receiving and distribution device, a first address processing device and a second address processing device connected with the data receiving and distribution device through internal interfaces; the first address processing device is connected with the old platform FEP through a first external interface; the second address processing device is connected with the new platform FEP through a second external interface; and the data receiving and distribution device is connected with each specialty through a third external interface. When the system receives and distributes data flow, IP address conversion is performed through the first address processing device and the second address processing device, so that each specialty can be connected with the new system and the old system in parallel, and if a fault is found during operation, the FEP can be quickly and directly connected with the specialty for fast recovery.

A modular and distributed architecture for data stream processing and analysis is described to incorporate data stream analytics capabilities, called Data Stream Analytics Service (DSAS) in the IoT / M2M service layer. Each service layer node hosting DSAS can be split into two independent modules, Stream Forwarder and Stream Analytics Engine. Stream Forwarder is a light weight processing modules that can be responsible for data preprocessing and routing. Stream Analytics Engine is responsible for performing actual analytics on the data stream. Separating the two functionalities enables the service layer nodes to efficiently distribute stream analytics tasks across multiple nodes.