107 results about "Distributive data processing" patented technology

The invention relates to the field of database reading, and particularly discloses an efficient database reading method based on multi-level cache optimization and dynamic index fragment.The efficient database reading method comprises the steps that high-frequency query data is accurately recognized through a sliding window algorithm, a dynamic cache loading and preheating strategy is implemented, query delay is effectively shortened, and the database burden is relieved; intelligent distribution of index fragments is realized by applying a consistent Hash algorithm, and the query efficiency is greatly improved in cooperation with distributed query routing and transaction processing optimization; the query load balancing ensures stable and efficient operation of the system in a high-concurrency scene by means of weighted polling and a minimum connection strategy in combination with real-time monitoring and an elastic capacity expansion and contraction mechanism. According to the method, the database access efficiency and the dynamic adaptive capacity are remarkably improved, and the method is particularly suitable for a large-scale distributed data processing environment.

The invention relates to the technical field of distributed data processing systems, and discloses a data processing system supporting full localization deployment and a compliance guarantee method thereof.The data processing system supporting full localization deployment comprises an audit event collection module used for collecting and storing processing event hash; the event organization module is used for realizing logic time sequence management of events through topological sorting and causal relationship marking; the hash chain construction module is used for constructing a local-to-global double-layer hash chain by utilizing the constructed directed acyclic graph and event hash; the increment synchronization module is used for executing increment synchronization based on Hash tree difference identification; the conflict solving module is used for detecting and solving event conflicts in the distributed environment by applying a version vector technology aiming at data differences found in the incremental synchronization process; according to the method, the lightweight linear hash chain is used for replacing a traditional binary tree structure, high-efficiency and low-cost audit chain maintenance is realized, and the data integrity is guaranteed in an unstable network environment.

This disclosure relates to quantum computing systems including a quantum processor that implements one or more operations on multiple qubits and a distributed data processing system programmed to perform calculations to determine control sequences that, when applied to the quantum processor, reduces decoherence, decoherence-induced errors, and control-imperfection-induced errors on the one or more operations on the multiple qubits. A user interface device remote from the distributed data processing system receives from a user of the quantum processor characteristics of the quantum processor including operational constraints and / or desired performance, and sends the characteristics to the data processing system to cause the data processing system to perform the calculations to determine the control sequence based on the characteristics.

The invention provides a local knowledge base automatic construction system based on multi-source acquisition and distributed computing, and relates to the technical field of computer systems. The system comprises a multi-source data collector and a distributed data processing and analyzing module, wherein the multi-source data collector realizes access, data collection, data transmission and incremental updating of multi-source data, and generates an incremental change record; and the distributed data processing and analyzing module comprises a plurality of distributed nodes and is used for carrying out distributed processing and analysis on data files according to incremental change records on the data collected by the multi-source data collector and establishing a local knowledge base. According to the system, automatic construction of the local knowledge base can be realized, the knowledge base construction efficiency is improved, the knowledge base construction period is shortened, the knowledge data freshness is ensured, and the data quality is improved.

A system and method are disclosed for generating a privacy-preserving data corpus for Artificial Intelligence (AI) training. The system comprises a relying partner (RP) computing environment and a trusted, independent identity provider (IdP) computing system. Upon a user authentication request, the IdP provides the RP with only a PII-free, persistent pseudonymous identifier (gUserID) for the user. Any authentication artifacts containing Personally Identifiable Information (PII), such as an OAuth token, are programmatically neutralized by the IdP. This is achieved by generating a transient public-private encryption key pair, immediately destroying the private key, and encrypting the PII-laden artifact with the remaining public key, rendering the PII therein permanently irrecoverable. This enforcement of “PII unknowability” at the RP enables the aggregation of pseudonymous user data, linked by the persistent gUserID, from multiple independent RPs into a rich, cross-organizational corpus for AI training, without ever exposing user PII to the RP.

The invention relates to the technical field of industrial equipment monitoring, in particular to an industrial equipment remote monitoring and early warning method based on data fusion, which comprises a multi-dimensional parameter dynamic correlation analysis module, an edge side distributed data processing unit, a self-adaptive threshold generator and an abnormal state early warning mechanism. Through capturing a multi-parameter nonlinear relationship, distributed data processing, dynamic threshold adjustment and a hierarchical early warning strategy, the monitoring real-time performance and accuracy are significantly improved, and the false alarm rate and the missing report rate are reduced. The method can effectively solve the problems of long monitoring period, high abnormity identification delay and the like, is suitable for large-scale industrial scenes, and has a wide application prospect.

The invention discloses an edge intelligent computing platform device and a data processing method thereof, and belongs to the field of computers, and the device comprises a customized hardware layer and a system software layer. The hardware layer is integrated with a plurality of high-performance GPUs supporting full-precision calculation, a large-capacity memory and an NVMe flash disk to form a strong computing power and storage basis. A system software layer is integrated, such as a Ray distributed framework, and the key improvement of the system is that an NVMe flash disk is configured as overflow storage of a memory, when the memory is insufficient due to processing of mass data or large model reasoning tasks, intermediate data can be automatically overflowed to a high-speed flash memory, and system crash is avoided. Through deep collaborative optimization of hardware and software, the technical problem that a large model cannot be efficiently operated and big data cannot be processed due to resource limitation on an edge side is effectively solved, and full-stack AI reasoning and high-performance distributed data processing at the edge side are realized.

Systems and methods are provide for automatically constructing data lineage representations for distributed data processing pipelines. These data lineage representations (which are constructed and stored in a central repository shared by the multiple data processing sites) can be used to among other things, clone the distributed data processing pipeline for quality assurance or debugging purposes. Examples of the presently disclosed technology are able to construct data lineage representations for distributed data processing pipelines by (1) generating a hash content value for universally identifying each data artifact of the distributed data processing pipeline across the multiple processing stages / processing sites of the distributed data processing pipeline; and (2) creating an data processing pipeline abstraction hierarchy for associating each data artifact to input and output events for given executions of given data processing stages (performed by the multiple data processing sites).

The invention relates to the technical field of cold chain transportation environment control, and discloses a temperature and humidity control system for cold chain transportation, and the system comprises an environment modeling module which is used for constructing a heat conduction model of a cold chain space based on a random field theory and a finite element method, and outputting a stiffness matrix containing random parameters; the self-adaptive sensing network module is used for receiving the characteristic information of the stiffness matrix, dynamically optimizing sensor distribution positions, collecting multi-node temperature and humidity data and adding space-time labels; and the distributed data processing module is used for receiving the sensing data. According to the invention, through random field theoretical modeling and parameter-state joint estimation, the temperature field prediction precision is significantly improved and the system environment adaptability is enhanced; a distributed sensing network architecture and a heterogeneous calculation acceleration technology are combined, so that the cooperative control with optimal energy efficiency is realized while millisecond-level real-time response is ensured, and the performance bottleneck of a traditional cold chain temperature control system is comprehensively broken through.

The invention discloses a distributed data processing method and system based on cloud computing, and relates to the technical field of distributed data processing.The method comprises the steps that heterogeneous resource parameters of computing nodes are obtained through a cloud platform interface, and the heterogeneous resource parameters comprise computing power indexes, network topology distances, real-time energy consumption efficiency and cloud service pricing data; constructing a multi-dimensional node feature vector; according to the multi-dimensional node feature vector, a dynamic weight distribution model is adopted to calculate a fragment weight value of each node, the weight distribution model introduces a normalization coefficient of energy consumption efficiency and cloud service pricing, and to-be-processed data is divided into data fragments in direct proportion to the fragment weight values; generating an inter-node communication cost matrix based on the network topology distance and cloud service pricing data, and distributing the data fragments to target computing nodes by adopting a graph matching algorithm to minimize the total communication cost; and setting task priorities, and screening key tasks and non-key tasks.

The invention belongs to the technical field of engineering management, and provides a decoration supervision service quality optimization method and system based on eight-dimensional feature evaluation, and the method comprises the steps: collecting multi-source heterogeneous data related to supervision service through a distributed data processing platform, carrying out the integrated preprocessing and deep fusion of the multi-source heterogeneous data, and carrying out the optimization of the quality of decoration supervision service. Outputting an eight-dimensional feature vector based on a depth feature engineering framework, generating an evaluation report through causal reasoning, generating an improvement suggestion according to an evaluation result, performing real-time updating, pushing the evaluation result and the improvement suggestion to a supervisor through a mobile terminal, tracking an improvement progress, and forming closed-loop management. According to the invention, the supervision personnel are evaluated from eight dimensions including supervision efficiency, service quality and satisfaction, and supervision human efficiency, core influence factors of each dimension are accurately found out, evaluation and improvement directions are output, the supervision personnel are supervised and promoted to learn and promote through the mobile terminal, a professional high-efficiency service team is established, and the efficiency of the supervision personnel is improved. And new development opportunities and revolution power are brought to the decoration industry.

An electronic device, according to one embodiment, may: acquire idle computing power information for an electronic device group; determine a first external electronic device from among the electronic device group on the basis of the idle computing power information when an execution command for a function requiring distributed processing of data is received; transmit a request for first data processing for performing the function to the first external electronic device; and perform the function on the basis of a first result for the first data processing received from the first external electronic device and a second result for second data processing performed by the electronic device for the function.

A mechanism for controlling a function performed by an application running on a processing system of an electronic device. The processing system is configured to host an application programming interface, API, that facilitates access to information about static and / or dynamic values of communication networks over which the electronic device is able to communicate. The application makes use of the available information through the API to control the function(s) performed by the application.

The technical scheme of the invention discloses a multi-source data fusion and intelligent processing method for vehicle network interaction. According to the invention, a multi-source data fusion and intelligent processing architecture oriented to vehicle network interaction is constructed, and technologies of multi-source heterogeneous data access, standardized modeling, distributed data processing, machine learning prediction, optimization scheduling and the like are comprehensively applied; real-time information interaction and collaborative optimization control among multiple subjects such as an electric vehicle, a charging facility and a power grid dispatching system are realized. Compared with the prior art, the scheme disclosed by the invention has the advantages that the data processing capability, the prediction precision, the response speed and the system adaptability are remarkably improved, and the requirements of vehicle network interaction large-scale application can be better met.

The invention discloses a distributed data processing method and system for an intelligent factory, and relates to the technical field of communication data processing. Determining target equipment according to the original data; based on the centrality index of the target equipment in the factory equipment topological graph, calculating the centrality index to obtain a topological criticality score; analyzing the original data based on a preset rule base to obtain an event severity score; multiplying the topology criticality score by the event severity score to obtain a final priority; inserting the original data into a priority scheduling queue according to the final priority; the storage nodes with the resource loads lower than a preset health threshold value are summarized into a current available node pool; and taking out the target data with the highest final priority from the head of the priority scheduling queue, selecting a storage node with the lowest resource load from the current available node pool as a target storage node, and preferentially distributing the target data to the target storage node for processing. By implementing the technical scheme, priority processing of high-value data is ensured.

The invention discloses a distributed data processing method in a local area network based on a UDP (User Datagram Protocol), which comprises the following steps: a distributed monitoring node sends a broadcast to the local area network, and a computing node in the local area network reports own identity information to the distributed monitoring node after receiving the broadcast; after a user initiates a data processing task, the task is sent to a target computing node in a structured instruction form through a UDP, the structured instruction comprises a task ID input data path algorithm type parameter set and priority, and a distributed monitoring node dynamically allocates the task according to the comprehensive load score of each node; the computing node sends a heartbeat message to the distributed monitoring node according to a heartbeat cycle, wherein the heartbeat message comprises a node identifier, a current running state, a current task execution identifier and node resource occupation summary information; according to the method, lightweight and low-delay distributed task scheduling can be realized. And meanwhile, node fault detection and task automatic recovery are realized, manual intervention is avoided, and safe and reliable execution of the task is guaranteed.

The invention discloses an edge intelligent distributed data processing method, and relates to the field of data processing. The method comprises the following steps: acquiring state data of at least one first edge device; determining at least one data processing task according to the target demand, wherein the data processing task comprises a task type, a processing deadline and a resource demand of the data processing task; based on the task type, the processing deadline and the resource demand of each data processing task, determining the task importance of the corresponding data processing task; according to the importance degree of each task, the computing resource occupancy rate of each first edge device and the network bandwidth occupancy rate, respectively calculating the processing adaptation degree of each first edge device and each data processing task; determining the stability of each first edge device based on the historical failure rate and the operation duration of the first edge device; and determining a second edge device based on the stability and the processing adaptation degree of each first edge device, and processing the data processing task through the second edge device. The data processing efficiency is improved.

This invention discloses a concurrency control and traffic scheduling system for card data interaction, relating to the fields of Internet of Things communications and distributed data processing technology. By adding a semantic addressing intent field (PIF) to the IPv6 extension header, the system enables switches to complete parsing and direct traffic to SmartNICs within 16ns. The SmartNIC generates a time-slot micro-ledger with a 500µs cycle and pre-allocates transaction locks using the Edge-Lock Weaving algorithm. Backend nodes complete strongly consistent transaction submissions using a single round of RDMA using a fast two-phase commit protocol. Dynamic congestion management is achieved through a dual-layer token bucket combined with flight windows. This solution achieves nanosecond read and write speeds, sub-millisecond submissions, and highly available decentralized scalability, significantly improving latency, throughput, and fault tolerance.

Embodiments of the present disclosure provide a method, an electronic device, and a computer program product for distributed data processing. A method in one embodiment comprises obtaining an input for a data processing task based on a multi-head attention mechanism, the data processing task comprising a first subtask and a second subtask, the first subtask corresponding to a first attention head in the multi-head attention mechanism, and the second subtask corresponding to a second attention head in the multi-head attention mechanism. The method further comprises transmitting the input to a first dedicated computing resource and a second dedicated computing resource, the first dedicated computing resource corresponding to the first subtask, and the second dedicated computing resource corresponding to the second subtask, and performing the first subtask and the second subtask on the input for obtaining an output of the data processing task.

The invention discloses an intelligent garden data management method and system based on cloud computing, and relates to the crossing field of cloud computing technologies, and the method comprises the steps: collecting and preprocessing multi-dimensional garden data through a distributed terminal, and adding classification and space-time labels; using a hybrid cloud architecture to realize data security storage and collaborative management, and establishing a classification index; a deep learning model is trained based on historical data, an analysis result is output, a personalized maintenance scheme is generated, and dynamic optimization is performed through real-time feedback; secure data intercommunication between the platform and various terminals is realized by means of a multi-terminal collaborative interaction module, and data security is guaranteed by adopting encrypted transmission and authority control; monitoring and encryption protection are implemented in the whole process, a traceable log is generated and filed regularly, and continuous iterative upgrading is performed in combination with a model effect. The method has the advantages that based on the hybrid cloud architecture and distributed data processing, garden management is promoted to be transformed to data intelligent driving through full-process closed-loop optimization, and the management efficiency and the refinement level are greatly improved.

A system and method generate a dynamic and query adaptive Bloom filter for optimal joining of tables such as large tables in distributed data processing systems. A Bloom filter is dynamically constructed and tuned based on historical query patterns including frequency and selectivity. Metrics are collected from execution of an Adaptive Query Execution (AQE) based execution plan and execution of a Bloom filter-based execution plan generated from the constructed and tuned Bloom filter. A score measuring performance of the execution plans is generated from the metrics of each of the execution plans. The execution plan with a greater score is selected to optimally join the tables. The method implements the system.

The application relates to a TDOA positioning method based on distributed data processing, and aims to solve the problem of low target positioning accuracy under limited sensor resources, and provides a TDOA positioning method based on distributed data processing.The specific process of the TDOA positioning method based on distributed data processing is as follows: step one, constructing a TDOA positioning system model; step two, obtaining an optimal sensor pairing combination based on the TDOA positioning system model; and step three, estimating the position p of a target source to be positioned based on the optimal sensor pairing combination to obtain an estimated value of the position p of the target source to be positioned.The application is used in the field of TDOA positioning.

This invention discloses an intelligent sales system for electronic components based on demand forecasting and dynamic pricing, belonging to the field of distributed data processing technology. It includes a data acquisition module that collects microservice call chain data through a distributed link tracing system; a semantic compression module that receives high-dimensional tensors output by an external prediction model and compresses them using an autoencoder neural network to generate low-dimensional feature vectors that the pricing engine can directly consume; an early warning module that predicts the probability of future resource conflicts based on historical resource competition index sequences using a time-series prediction model; a scheduling module that constructs a coupling degree matrix of system variables and calculates coupling entropy, and executes resource scheduling strategies according to nonlinear scheduling decision factors; and a feedback optimization module that dynamically updates the early warning threshold by collecting performance data after scheduling. This invention significantly reduces system response latency and resource consumption.

The invention relates to the technical field of edge computing and distributed data processing, and discloses a multi-source data analysis and acquisition system and method for an automobile radar, and the system comprises the steps: dividing a safe reserved water level and a transaction quota water level in a shared memory pool of a radar on-chip system; the evaluation module calculates a perception difference degree between the radar and the visual frame, and triggers capture when the perception difference degree exceeds a threshold value; the coding module generates a standardized inventory index; the arbitration module queries a local bloom filter, executes speculative locking when the local bloom filter is not hit and the number of the to-be-determined memory blocks is lower than the quota water level, or executes fusing; and the settlement module sends the index to the cloud and dispatches the controller to transmit data according to a supply and demand saturation instruction fed back by the cloud. Through end-cloud cooperative verification and memory resource level-to-level management, the problem that vehicle-mounted high-frequency data acquisition and a driving safety function compete for resources is solved, and the effectiveness and safety of data acquisition are improved.

The invention discloses a distributed power operation state monitoring method. The method comprises the following steps that an edge server carries out distributed data processing on secondary equipment data information; the edge server carries out encryption processing on the distributed data processing result and transmits the result to the processing center; the processing center receives the distributed data processing result transmitted by the edge server, and generates a monitoring report based on the distributed data processing result; the distributed processing of the secondary equipment data information is realized through the edge server, the encrypted data information is transmitted to the processing center, the processing center generates the monitoring report based on the distributed data processing result, and the operation state of the power equipment can be monitored in real time.

To provide a distributed data processing delay factor analysis and improvement method, a device, and a computer program to a user.SOLUTION: In an analysis device, a method includes: a step of calculating simulation information for simulation of an application based on a log file for the application that performs distribution processing on data; a step of performing the simulation in a plurality of driving environments including an actual driving environment of the application and a virtual driving environment in which one or more delay factors are removed by using the simulation information; and a step of proceeding with analysis on the application based on a simulation result in the plurality of driving environments.SELECTED DRAWING: Figure 2

The application discloses a parallel decision system and method for distributed data processing. The system comprises: an initial logical node generation component, which generates an initial logical node topology; a direct neighborhood marking component, which traverses the initial logical node topology to obtain a direct neighborhood topology of a selected initial logical node and marks the obtained direct neighborhood topology one by one; a sub-neighborhood cost calculation component, which traverses each sub-neighborhood topology in the selected marked direct neighborhood topology, and calculates the cost sum of the sub-neighborhood topology under each random combination of the sub-neighborhood topology and the node cost sum of each initial logical node; a sub-neighborhood parallel decision component, which calculates the cost sum difference between the node cost sum of each initial logical node under all random combinations, and determines the sub-neighborhood parallel scheme of the sub-neighborhood topology; and a parallel decision termination component, which terminates the repeated traversal process under the condition that the parallel scheme of each direct neighborhood obtained by two consecutive traversals and the node cost sum assigned to each initial logical node remain unchanged.

This disclosure relates to quantum computing systems including a quantum processor that implements one or more operations on multiple qubits and a distributed data processing system programmed to perform calculations to determine control sequences that, when applied to the quantum processor, reduces decoherence, decoherence-induced errors, and control-imperfection-induced errors on the one or more operations on the multiple qubits. A user interface device remote from the distributed data processing system receives from a user of the quantum processor characteristics of the quantum processor including operational constraints and / or desired performance, and sends the characteristics to the data processing system to cause the data processing system to perform the calculations to determine the control sequence based on the characteristics.

This application discloses a method and related products for asynchronous order data processing, applicable to the fields of e-commerce and distributed data processing technology. The method includes: retrieving order data and sending the corresponding order messages to a customer queue; processing the order messages in the customer queue based on a first distributed lock, saving the customer information corresponding to the order messages, and sending the order messages to an order queue; processing the order messages in the order queue based on a second distributed lock, saving the order information corresponding to the order messages, and sending the order messages to a notification queue; and processing the order messages in the notification queue and saving the notification information corresponding to the order messages. Thus, by asynchronously processing order data based on three independent queues, business logic decoupling and resource isolation are achieved, mutual blocking is avoided, and the stability of data processing is improved.