16 results about "Distributed data flow" patented technology

The invention relates to the technical field of network communication, in particular to a multipath transmission load balancing method and system based on network state analysis. The method comprises the steps of collecting network state information of each transmission link from network equipment, calculating and obtaining the weight of each transmission link according to the collected network state information, setting a flow distribution strategy according to the weight obtained by calculation of each transmission link, and distributing data flow to each transmission link in proportion, and collecting the network state information of each transmission link in real time, updating the weight of each transmission link according to the change of each parameter in the network state information, performing threshold judgment on the updated weight, and dynamically adjusting the flow distribution strategy according to the weight meeting the threshold requirement. According to the method, the defects of a traditional static strategy can be effectively overcome, link congestion or idle is avoided, high efficiency and stability of data transmission are guaranteed, and finally the requirement of a user for high-quality network service is met.

The embodiment of the invention discloses a product pushing method, electronic equipment, a storage medium and a program product, and relates to the technical field of financial science and technology, and the method comprises the steps: obtaining behavior data of a user in a target application before the current moment from a distributed publishing and subscribing system through a distributed data flow engine, obtaining real-time behavior data, and transmitting the real-time behavior data to a server; obtaining offline behavior data of the user from the data warehouse to obtain historical behavior data; fusing the real-time behavior data and the historical behavior data to obtain user behavior data; determining a pushing rule satisfied by the user behavior data from a preset pushing rule base to obtain a plurality of candidate rules; obtaining a static feature and a dynamic feature of each candidate rule, and determining a priority score of the corresponding candidate rule based on the static feature and the dynamic feature; and the optimal rule is determined based on the priority score of each candidate rule, and the target product corresponding to the optimal rule is pushed to the user terminal of the user, so that the timeliness and accuracy of product pushing are improved.

The invention discloses a real-time processing method and system for distributed data streams, and particularly relates to the technical field of distributed data processing, and the method comprises the following steps: collecting multi-source data streams and marking priorities, combining data types and priorities to carry out adaptive segmentation, carrying out load balancing distribution on processing units, and resolving data conflicts; and incremental integration and caching of hotspot data are carried out, and full-link tracing and dynamic tuning are carried out. The system comprises five modules, namely an acquisition labeling module, a segmentation module, a parallel processing module, an integration optimization module and a tracing tuning module. According to the real-time processing method and system for the distributed data flow, priority processing of core data is achieved, data consistency is guaranteed, the resource utilization rate and the system maintainability are improved, and the processing requirements for high real-time performance and high reliability are met.

The invention discloses an identity authentication method and device in a distributed data circulation environment, equipment and a medium, and relates to the technical field of computers. Comprising the following steps: acquiring an identity registration request of an access subject through a regional function node, verifying real-name identity information of the access subject, and returning a corresponding identity registration result to the access subject based on a verification result of the real-name identity information; obtaining an access connector registration request of an access main body through a regional function node, and issuing a trusted identity certificate to the access main body and a target access connector according to an information auditing result of the target access connector; and if a login request of the access main body is received through the regional function node, the access main body is accessed to a preset service processing platform through the target access connector based on the trusted identity certificate to complete an identity authentication process. Therefore, cross-domain identity mutual recognition and hierarchical authorization can be realized, and basic identity guarantee is provided for safe, efficient and compliant circulation of data elements.

The application discloses a kind of distributed database synchronization method and system and device and medium, it is related to data synchronization technical field, the method includes the following steps: the first parameter information of configuration distributed database;Configuration database management system second parameter information;Configuration satisfies the third parameter information of distributed data stream processing unit operation;The first parameter information, the second parameter information and the third parameter information are parsed and verified, and parsing verification result is obtained;Based on the parsing verification result, create structured query language and generate real-time warehousing code script;The structured query language and the real-time warehousing code script are uploaded to the distributed data stream processing unit release and execute synchronization task and obtain data synchronization result.The application adopts distributed data stream processing unit to execute data synchronization task, improves synchronization efficiency, significantly reduces distributed database full data synchronization time.

Techniques for log storage in distributed data streaming systems are described. A cluster of brokers receive log records from publishers and send log records to subscribers. The log is represented as a group of segments, each segment subdivided into chunks. Metadata describes the log structure. Log records are stored in chunks at least in a remote storage location shared amongst the brokers in the cluster.

The invention provides a data stream adaptive scheduling method, system and device and a medium, and belongs to the technical field of distributed data stream processing. The method comprises the following steps: acquiring NiFi system operation index data, and extracting standardized features by using feature engineering; a machine learning model combination comprising a neural network, a gradient elevator and a reinforcement learning model is constructed for prediction analysis, and an execution time prediction value, a failure probability prediction value and scheduling strategy parameters are generated; dynamically adjusting system configuration parameters based on the prediction result; risk assessment and graded alarm are realized through a multi-stage rule engine; performing real-time state display and configuration management by adopting a visual engine; and finally, an optimization suggestion report is generated through the data analysis platform. According to the invention, adaptive scheduling and intelligent operation and maintenance of the data stream processing system are realized, and the resource utilization efficiency and the operation stability of the system are effectively improved.

The invention relates to the technical field of computer architecture, and provides a computer system and a data processing method, and the system comprises the steps that an on-chip network distributes data traffic according to an address interleaving strategy; the data link module comprises a plurality of data link units, and each data link unit comprises a plurality of data links; the network transmission module comprises a plurality of network transmission layers, the plurality of network transmission layers are in communication connection with the network-on-chip, each network transmission layer is in communication connection with the plurality of data links in one data link unit, and the network transmission layers receive first data sent by the network-on-chip and send the first data to the network-on-chip; distributing the first data to at least one data link in the data link unit according to a preset scheduling strategy; and transmitting second data received from at least one data link in the data link unit to the network-on-chip. According to the invention, through a one-to-many hierarchical link architecture, when a certain link fails, the link can be switched to other links, so that the stability and reliability of the system are enhanced.

The invention provides a distributed vehicle sensor system anomaly detection method based on streaming data. The system is composed of a coordinator Cord and sensor systems of m vehicles. Comprising the following five steps: step 1, a cold start stage; step 2, maintaining a local accumulative Frobenius norm Fi and a global accumulative Frobenius norm; step 3, updating a local matrix sketch D through a matrix sketch algorithm on a distributed sliding window, and returning two matrixes; 4, sending a matrix to the coordinator Cord through a channel, and notifying the coordinator Cord to update a global test window matrix sketch K2; 5, the vector ait is used for updating an index histogram matrix sketch mEH, if the timestamp et of an element e with the minimum timestamp in the updated mEH is smaller than the size W of a time subtraction window at the current time, the site Si sends the element e to the coordinator Cord through a channel, and the coordinator is notified to execute expiration discarding operation on the element. Therefore, the effect of efficiently identifying the abnormal behavior in the distributed data stream of the vehicle sensor system is achieved.

The invention provides a data traffic intelligent scheduling method and device, electronic equipment and a storage medium. The method comprises the following steps: sending a management message to a network interconnection device through a plurality of links, wherein the management message is a message used for detecting interconnection and interworking states of the plurality of links corresponding to the network interconnection device; determining link quality parameters corresponding to the plurality of links according to a received response message fed back by the network interconnection equipment; and scheduling and distributing data traffic corresponding to the plurality of links based on the link quality parameters. According to the method and the device, the link quality of the plurality of links can be detected on the premise of not increasing the network load, so that the possible upper limit of data transmission of each link can be predicted in advance, the data traffic distributed to each link is adjusted, and the data transmission quality is ensured.

A modular and distributed architecture for data stream processing and analysis is described to incorporate data stream analytics capabilities, called Data Stream Analytics Service (DSAS) in the IoT / M2M service layer. Each service layer node hosting DSAS can be split into two independent modules, Stream Forwarder and Stream Analytics Engine. Stream Forwarder is a light weight processing modules that can be responsible for data preprocessing and routing. Stream Analytics Engine is responsible for performing actual analytics on the data stream. Separating the two functionalities enables the service layer nodes to efficiently distribute stream analytics tasks across multiple nodes.

The invention provides a product information pushing method, device, medium and system for a mobile banking, and the method comprises the steps: collecting user transaction behavior data of the mobile banking through employing a data processing pipeline tool under the condition of obtaining user authorization, and caching the user transaction behavior data through employing a distributed message caching queue; summarizing the cached user transaction behavior data by adopting a distributed data flow calculation engine according to a preset dimension rule to obtain a summarizing result, and storing the summarizing result into a column database; and querying the data in the column database by adopting the query statement to obtain user behavior data and product transaction information, processing the user behavior data and the product transaction information to obtain potential product information, and pushing the potential product information. Therefore, the problem that in the prior art, a platform can only query the user funnel diagram to achieve a display function, but cannot push the product information for the customer loss rate in the user funnel is solved.

The application provides an interface parallel distribution method and system for comprehensive monitoring system reconstruction, and the system comprises an interface parallel distribution system for connecting a new platform, an old platform and each specialty; the interface parallel distribution system comprises a data receiving and distribution device, a first address processing device and a second address processing device connected with the data receiving and distribution device through internal interfaces; the first address processing device is connected with the old platform FEP through a first external interface; the second address processing device is connected with the new platform FEP through a second external interface; and the data receiving and distribution device is connected with each specialty through a third external interface. When the system receives and distributes data flow, IP address conversion is performed through the first address processing device and the second address processing device, so that each specialty can be connected with the new system and the old system in parallel, and if a fault is found during operation, the FEP can be quickly and directly connected with the specialty for fast recovery.

A modular and distributed architecture for data stream processing and analysis is described to incorporate data stream analytics capabilities, called Data Stream Analytics Service (DSAS) in the IoT / M2M service layer. Each service layer node hosting DSAS can be split into two independent modules, Stream Forwarder and Stream Analytics Engine. Stream Forwarder is a light weight processing modules that can be responsible for data preprocessing and routing. Stream Analytics Engine is responsible for performing actual analytics on the data stream. Separating the two functionalities enables the service layer nodes to efficiently distribute stream analytics tasks across multiple nodes.