154 results about "Distributive data processing" patented technology

The invention relates to the technical field of data transfer and verification, in particular to a business data testing method and system and a storage medium. The method comprises the following steps: acquiring an original service data stream; deploying a supervision submission shadow processing sandbox based on the original business data flow to obtain a data isolation verification environment; configuring an alliance block chain server based on the original service data flow, and constructing a distributed data processing traceability platform; based on the distributed data processing traceability platform, performing data processing proof generation by using the original service data stream to obtain a data processing hash chain; configuring differential privacy protection processor parameters based on the data isolation verification environment to obtain a privacy protection test data generator; a privacy protection test data generator is trained to generate a synthetic test data set. By combining the alliance block chain and the differential privacy technology, a high-credibility, automatic and traceable data testing system is constructed, and the data security, the verification efficiency and the system consistency are remarkably improved.

The invention discloses a remote centralized control distributed data processing and analysis method based on a JMS, and the method comprises the steps: achieving the message routing and resource scheduling through the real-time monitoring of the state of a system, reinforcement of learning decision, deep learning optimization and multi-dimensional message distribution; dynamically adjusting message flow rate and queue length based on real-time network conditions, node load and flow fluctuation; through edge node intelligent task scheduling, distributed collaboration of calculation and storage is realized; adaptive compression algorithm selection is carried out by using deep learning, and the consumption of message transmission bandwidth is reduced; non-tampering storage of messages is realized through a block chain technology; based on a container arrangement technology, dynamic expansion and contraction of computing resources are realized; and dynamic resource scheduling and optimization decision are realized by utilizing reinforcement learning and multi-dimensional monitoring data. According to the invention, the bottleneck problem of the traditional message transmission system is overcome, and the system can efficiently and reliably process mass data in a high-concurrency and high-throughput scene.

The invention relates to the technical field of electric meter online monitoring, in particular to an electric energy data intelligent acquisition terminal based on federal learning. The data acquisition and preprocessing module is used for acquiring electric energy data in a power grid in real time and carrying out de-noising and normalization preprocessing on the electric energy data; the model training module based on federated learning is used for training a neural network model for the preprocessed electric energy data by using local data so as to obtain a local model; the homomorphic encryption algorithm application module is used for encrypting and updating data and model parameters in a local data training process by adopting a homomorphic encryption algorithm; and the global model updating module is used for sending the neural network model subjected to multiple times of local data training and encryption updating to a central server, and optimizing and updating the neural network model through a weighted federated average aggregation algorithm. According to the invention, distributed data processing and machine learning technologies are combined to realize intelligent data analysis of electric power data, so that the intelligent level of electric power monitoring and fault early warning is improved.

The invention relates to the field of database reading, and particularly discloses an efficient database reading method based on multi-level cache optimization and dynamic index fragment.The efficient database reading method comprises the steps that high-frequency query data is accurately recognized through a sliding window algorithm, a dynamic cache loading and preheating strategy is implemented, query delay is effectively shortened, and the database burden is relieved; intelligent distribution of index fragments is realized by applying a consistent Hash algorithm, and the query efficiency is greatly improved in cooperation with distributed query routing and transaction processing optimization; the query load balancing ensures stable and efficient operation of the system in a high-concurrency scene by means of weighted polling and a minimum connection strategy in combination with real-time monitoring and an elastic capacity expansion and contraction mechanism. According to the method, the database access efficiency and the dynamic adaptive capacity are remarkably improved, and the method is particularly suitable for a large-scale distributed data processing environment.

The invention is applicable to the field of equipment data collection and processing, and provides a field video equipment control method which comprises the following steps: acquiring a network identifier of video equipment; according to the network identifier, mutually connecting the video devices with the same network identifier, establishing a data transmission network, and obtaining a theoretical monitoring range and an actual observable area of the video devices with the same network identifier to obtain a monitorable area; generating a video equipment control instruction according to the monitorable area; according to the monitorable area, splicing images in the monitorable area to obtain a spliced image; according to a video equipment control instruction and a spliced image, video equipment is deployed in a unified manner, through unified task deployment and distributed data processing, resources of a plurality of cameras are fully utilized, species information at different angles and distances is captured, more comprehensive observation data is provided, the monitoring efficiency is improved, and the coverage range is expanded. The system is suitable for large-range species monitoring requirements, and can timely deploy tasks according to feedback data, change a shooting angle or increase monitoring frequency and the like.

The invention relates to the technical field of distributed data processing systems, and discloses a data processing system supporting full localization deployment and a compliance guarantee method thereof.The data processing system supporting full localization deployment comprises an audit event collection module used for collecting and storing processing event hash; the event organization module is used for realizing logic time sequence management of events through topological sorting and causal relationship marking; the hash chain construction module is used for constructing a local-to-global double-layer hash chain by utilizing the constructed directed acyclic graph and event hash; the increment synchronization module is used for executing increment synchronization based on Hash tree difference identification; the conflict solving module is used for detecting and solving event conflicts in the distributed environment by applying a version vector technology aiming at data differences found in the incremental synchronization process; according to the method, the lightweight linear hash chain is used for replacing a traditional binary tree structure, high-efficiency and low-cost audit chain maintenance is realized, and the data integrity is guaranteed in an unstable network environment.

The invention relates to the technical field of artificial intelligence, in particular to a data processing method based on artificial intelligence. Comprising the following steps: a) a data preprocessing module: performing primary processing on input data, including but not limited to filling vacant values, identifying and processing abnormal values, removing repeated data and unifying data formats, so as to ensure the quality and consistency of the data; b) a feature selection and dimension reduction module: screening out key features from original data and reducing data dimensions by using an automatic feature selection method; c) a data security and privacy protection module; d) a distributed data processing and storage module; e) an intelligent model training and evaluation module; and f) a feedback and optimization module. The invention provides the artificial intelligence-based data processing method which can improve the data preprocessing capability, optimize the feature selection and dimension reduction effect, enhance the data security and privacy protection capability, improve the distributed data processing and storage efficiency, improve the intelligent model training and evaluation performance and perfect the feedback and optimization mechanism.

The invention discloses a GNSS network RTK base station distributed data processing method and device, belongs to the technical field of satellite navigation and positioning, and is used for solving key problems of base station networking, baseline solution, error correction calculation, model establishment and the like in a large GNSS network RTK system. The method is especially suitable for efficient calculation, task allocation and dynamic load balancing processing of a large-scale base station network. The method comprises the following steps: 1) dynamically networking a core server, determining a baseline to be solved, generating a baseline solving task and establishing an error model; (2) a plurality of data processing servers solve all baselines in parallel, and baseline error correction numbers are extracted; (3) each base station calculates relative corrections of other stations relative to the base station according to the related baseline corrections, and establishes an error model, and (4) dynamically allocates a baseline resolving task and an error modeling task. Through multi-thread and distributed parallel computing, the task processing efficiency and the resource utilization rate of a large-scale base station network are remarkably improved.

This disclosure relates to quantum computing systems including a quantum processor that implements one or more operations on multiple qubits and a distributed data processing system programmed to perform calculations to determine control sequences that, when applied to the quantum processor, reduces decoherence, decoherence-induced errors, and control-imperfection-induced errors on the one or more operations on the multiple qubits. A user interface device remote from the distributed data processing system receives from a user of the quantum processor characteristics of the quantum processor including operational constraints and / or desired performance, and sends the characteristics to the data processing system to cause the data processing system to perform the calculations to determine the control sequence based on the characteristics.

The invention provides a local knowledge base automatic construction system based on multi-source acquisition and distributed computing, and relates to the technical field of computer systems. The system comprises a multi-source data collector and a distributed data processing and analyzing module, wherein the multi-source data collector realizes access, data collection, data transmission and incremental updating of multi-source data, and generates an incremental change record; and the distributed data processing and analyzing module comprises a plurality of distributed nodes and is used for carrying out distributed processing and analysis on data files according to incremental change records on the data collected by the multi-source data collector and establishing a local knowledge base. According to the system, automatic construction of the local knowledge base can be realized, the knowledge base construction efficiency is improved, the knowledge base construction period is shortened, the knowledge data freshness is ensured, and the data quality is improved.

A system and method are disclosed for generating a privacy-preserving data corpus for Artificial Intelligence (AI) training. The system comprises a relying partner (RP) computing environment and a trusted, independent identity provider (IdP) computing system. Upon a user authentication request, the IdP provides the RP with only a PII-free, persistent pseudonymous identifier (gUserID) for the user. Any authentication artifacts containing Personally Identifiable Information (PII), such as an OAuth token, are programmatically neutralized by the IdP. This is achieved by generating a transient public-private encryption key pair, immediately destroying the private key, and encrypting the PII-laden artifact with the remaining public key, rendering the PII therein permanently irrecoverable. This enforcement of “PII unknowability” at the RP enables the aggregation of pseudonymous user data, linked by the persistent gUserID, from multiple independent RPs into a rich, cross-organizational corpus for AI training, without ever exposing user PII to the RP.

The invention provides a cement industry internet-of-things intelligent logistics system, which comprises the following steps of: according to standardized sales order data and delivery order data, judging the relevance between the sales order data and the delivery order data by adopting a decision tree algorithm, establishing a mapping relationship, and realizing information connection between sales and logistics links; vehicle positioning data and loading and unloading confirmation information are obtained, a real-time tracking record of the whole logistics process is generated through data fusion processing, and a visual track is generated in combination with map information; for mass logistics data, a distributed data processing platform is established, the execution efficiency of logistics tasks is analyzed by adopting a clustering algorithm, abnormal tasks are identified, and intelligent early warning signals are generated; and according to the intelligent early warning signal, the priority of the logistics task and the vehicle scheduling strategy are adjusted, the distribution of logistics resources is optimized, and the overall logistics efficiency is improved.

The invention relates to the technical field of e-commerce, in particular to a distributed data processing system based on big data, and the system comprises an internal data collection end which is used for collecting user transaction data, commodity data and behavior logs of an e-commerce platform in real time; the external data acquisition end is used for acquiring consumer credit data and social network behavior data of the third-party platform through an API (Application Program Interface); the data classification module is used for carrying out initial classification on the collected data by adopting a multi-level label system and generating classification labels with weight coefficients; the data screening module is used for dynamically filtering the classified data according to a preset threshold condition, and the threshold condition comprises data timeliness, integrity and confidence; the data error correction module is used for correcting the labels of the misclassification data set and feeding back the corrected data to the input end of the classification module; and the plurality of distributed data analysis centers generate 16 user portrait dimensions from A1 to D4, and the analysis centers execute data modeling of the corresponding dimensions in parallel.

The invention relates to the technical field of industrial equipment monitoring, in particular to an industrial equipment remote monitoring and early warning method based on data fusion, which comprises a multi-dimensional parameter dynamic correlation analysis module, an edge side distributed data processing unit, a self-adaptive threshold generator and an abnormal state early warning mechanism. Through capturing a multi-parameter nonlinear relationship, distributed data processing, dynamic threshold adjustment and a hierarchical early warning strategy, the monitoring real-time performance and accuracy are significantly improved, and the false alarm rate and the missing report rate are reduced. The method can effectively solve the problems of long monitoring period, high abnormity identification delay and the like, is suitable for large-scale industrial scenes, and has a wide application prospect.

The invention discloses an edge intelligent computing platform device and a data processing method thereof, and belongs to the field of computers, and the device comprises a customized hardware layer and a system software layer. The hardware layer is integrated with a plurality of high-performance GPUs supporting full-precision calculation, a large-capacity memory and an NVMe flash disk to form a strong computing power and storage basis. A system software layer is integrated, such as a Ray distributed framework, and the key improvement of the system is that an NVMe flash disk is configured as overflow storage of a memory, when the memory is insufficient due to processing of mass data or large model reasoning tasks, intermediate data can be automatically overflowed to a high-speed flash memory, and system crash is avoided. Through deep collaborative optimization of hardware and software, the technical problem that a large model cannot be efficiently operated and big data cannot be processed due to resource limitation on an edge side is effectively solved, and full-stack AI reasoning and high-performance distributed data processing at the edge side are realized.

Systems and methods are provide for automatically constructing data lineage representations for distributed data processing pipelines. These data lineage representations (which are constructed and stored in a central repository shared by the multiple data processing sites) can be used to among other things, clone the distributed data processing pipeline for quality assurance or debugging purposes. Examples of the presently disclosed technology are able to construct data lineage representations for distributed data processing pipelines by (1) generating a hash content value for universally identifying each data artifact of the distributed data processing pipeline across the multiple processing stages / processing sites of the distributed data processing pipeline; and (2) creating an data processing pipeline abstraction hierarchy for associating each data artifact to input and output events for given executions of given data processing stages (performed by the multiple data processing sites).

The invention relates to the technical field of distributed data processing, and discloses a distributed financial data processing system. The system comprises a multi-source data acquisition module, a heterogeneous data integration module, a distributed calculation verification module, a service flow modeling module, a resource scheduling module and the like. The multi-source data acquisition module captures multi-dimensional financial transaction data in real time by using a distributed edge computing node, and distributes an acquisition path through a dynamic load balancing algorithm; the heterogeneous data integration module analyzes the cross-platform data format and generates a standardized data stream; the distributed calculation verification module constructs a credible verification model based on a hybrid consensus mechanism; the business flow modeling module predicts a risk conduction path; and the resource scheduling module optimizes resource allocation. The system effectively solves the problems of a traditional financial data processing system in the aspects of data acquisition, integration, calculation verification and the like, improves the data processing efficiency, the risk prediction capability and the resource utilization efficiency, and guarantees the safety and compliance of financial transactions.

The invention relates to the technical field of cold chain transportation environment control, and discloses a temperature and humidity control system for cold chain transportation, and the system comprises an environment modeling module which is used for constructing a heat conduction model of a cold chain space based on a random field theory and a finite element method, and outputting a stiffness matrix containing random parameters; the self-adaptive sensing network module is used for receiving the characteristic information of the stiffness matrix, dynamically optimizing sensor distribution positions, collecting multi-node temperature and humidity data and adding space-time labels; and the distributed data processing module is used for receiving the sensing data. According to the invention, through random field theoretical modeling and parameter-state joint estimation, the temperature field prediction precision is significantly improved and the system environment adaptability is enhanced; a distributed sensing network architecture and a heterogeneous calculation acceleration technology are combined, so that the cooperative control with optimal energy efficiency is realized while millisecond-level real-time response is ensured, and the performance bottleneck of a traditional cold chain temperature control system is comprehensively broken through.

The invention discloses a distributed data processing method and system based on cloud computing, and relates to the technical field of distributed data processing.The method comprises the steps that heterogeneous resource parameters of computing nodes are obtained through a cloud platform interface, and the heterogeneous resource parameters comprise computing power indexes, network topology distances, real-time energy consumption efficiency and cloud service pricing data; constructing a multi-dimensional node feature vector; according to the multi-dimensional node feature vector, a dynamic weight distribution model is adopted to calculate a fragment weight value of each node, the weight distribution model introduces a normalization coefficient of energy consumption efficiency and cloud service pricing, and to-be-processed data is divided into data fragments in direct proportion to the fragment weight values; generating an inter-node communication cost matrix based on the network topology distance and cloud service pricing data, and distributing the data fragments to target computing nodes by adopting a graph matching algorithm to minimize the total communication cost; and setting task priorities, and screening key tasks and non-key tasks.

The invention belongs to the technical field of engineering management, and provides a decoration supervision service quality optimization method and system based on eight-dimensional feature evaluation, and the method comprises the steps: collecting multi-source heterogeneous data related to supervision service through a distributed data processing platform, carrying out the integrated preprocessing and deep fusion of the multi-source heterogeneous data, and carrying out the optimization of the quality of decoration supervision service. Outputting an eight-dimensional feature vector based on a depth feature engineering framework, generating an evaluation report through causal reasoning, generating an improvement suggestion according to an evaluation result, performing real-time updating, pushing the evaluation result and the improvement suggestion to a supervisor through a mobile terminal, tracking an improvement progress, and forming closed-loop management. According to the invention, the supervision personnel are evaluated from eight dimensions including supervision efficiency, service quality and satisfaction, and supervision human efficiency, core influence factors of each dimension are accurately found out, evaluation and improvement directions are output, the supervision personnel are supervised and promoted to learn and promote through the mobile terminal, a professional high-efficiency service team is established, and the efficiency of the supervision personnel is improved. And new development opportunities and revolution power are brought to the decoration industry.

The present application provides a distributed data processing method, system, device and storage medium, which relates to the field of distributed data processing technology, obtains industrial data packets to be processed in a distributed system, performs data compression on the industrial data packets to be processed, and then obtains multiple data compression slices; extracts the industrial data covariates corresponding to each data compression slice, thereby determining the distribution density disorder of each data compression slice, and then determining the data covariance threshold value of each data compression slice; determines the potential abnormal data of each data compression slice based on the corresponding data covariance threshold value, and determines the de-differentiated data packets in the distributed processing of industrial data through the potential abnormal data of each data compression slice; stores the de-differentiated data packets to complete the distributed processing of industrial data. The present application can set a dynamic adaptive threshold that takes into account both global accuracy and local sensitivity to improve the accuracy of anomaly elimination in distributed data processing.

An electronic device, according to one embodiment, may: acquire idle computing power information for an electronic device group; determine a first external electronic device from among the electronic device group on the basis of the idle computing power information when an execution command for a function requiring distributed processing of data is received; transmit a request for first data processing for performing the function to the first external electronic device; and perform the function on the basis of a first result for the first data processing received from the first external electronic device and a second result for second data processing performed by the electronic device for the function.

A mechanism for controlling a function performed by an application running on a processing system of an electronic device. The processing system is configured to host an application programming interface, API, that facilitates access to information about static and / or dynamic values of communication networks over which the electronic device is able to communicate. The application makes use of the available information through the API to control the function(s) performed by the application.

The technical scheme of the invention discloses a multi-source data fusion and intelligent processing method for vehicle network interaction. According to the invention, a multi-source data fusion and intelligent processing architecture oriented to vehicle network interaction is constructed, and technologies of multi-source heterogeneous data access, standardized modeling, distributed data processing, machine learning prediction, optimization scheduling and the like are comprehensively applied; real-time information interaction and collaborative optimization control among multiple subjects such as an electric vehicle, a charging facility and a power grid dispatching system are realized. Compared with the prior art, the scheme disclosed by the invention has the advantages that the data processing capability, the prediction precision, the response speed and the system adaptability are remarkably improved, and the requirements of vehicle network interaction large-scale application can be better met.

The invention discloses a distributed data processing method and system for an intelligent factory, and relates to the technical field of communication data processing. Determining target equipment according to the original data; based on the centrality index of the target equipment in the factory equipment topological graph, calculating the centrality index to obtain a topological criticality score; analyzing the original data based on a preset rule base to obtain an event severity score; multiplying the topology criticality score by the event severity score to obtain a final priority; inserting the original data into a priority scheduling queue according to the final priority; the storage nodes with the resource loads lower than a preset health threshold value are summarized into a current available node pool; and taking out the target data with the highest final priority from the head of the priority scheduling queue, selecting a storage node with the lowest resource load from the current available node pool as a target storage node, and preferentially distributing the target data to the target storage node for processing. By implementing the technical scheme, priority processing of high-value data is ensured.

The invention discloses a distributed data processing method in a local area network based on a UDP (User Datagram Protocol), which comprises the following steps: a distributed monitoring node sends a broadcast to the local area network, and a computing node in the local area network reports own identity information to the distributed monitoring node after receiving the broadcast; after a user initiates a data processing task, the task is sent to a target computing node in a structured instruction form through a UDP, the structured instruction comprises a task ID input data path algorithm type parameter set and priority, and a distributed monitoring node dynamically allocates the task according to the comprehensive load score of each node; the computing node sends a heartbeat message to the distributed monitoring node according to a heartbeat cycle, wherein the heartbeat message comprises a node identifier, a current running state, a current task execution identifier and node resource occupation summary information; according to the method, lightweight and low-delay distributed task scheduling can be realized. And meanwhile, node fault detection and task automatic recovery are realized, manual intervention is avoided, and safe and reliable execution of the task is guaranteed.

The present disclosure relates to a quantum computing system, the quantum computing system comprising: a quantum processor that implements one or more operations on a plurality of qubits; and a distributed data processing system that is programmed to perform calculations to determine a control sequence that, when applied to the quantum processor, reduces decoherence, decoherence-induced errors, and control-defect-induced errors of the one or more operations on the plurality of qubits. A user interface device remote from the distributed data processing system receives, from a user of the quantum processor, characteristics of the quantum processor including operation constraints and / or desired performance, and sends the characteristics to the data processing system to cause the data processing system to perform calculations to determine a control sequence based on the characteristics.

The invention discloses an edge intelligent distributed data processing method, and relates to the field of data processing. The method comprises the following steps: acquiring state data of at least one first edge device; determining at least one data processing task according to the target demand, wherein the data processing task comprises a task type, a processing deadline and a resource demand of the data processing task; based on the task type, the processing deadline and the resource demand of each data processing task, determining the task importance of the corresponding data processing task; according to the importance degree of each task, the computing resource occupancy rate of each first edge device and the network bandwidth occupancy rate, respectively calculating the processing adaptation degree of each first edge device and each data processing task; determining the stability of each first edge device based on the historical failure rate and the operation duration of the first edge device; and determining a second edge device based on the stability and the processing adaptation degree of each first edge device, and processing the data processing task through the second edge device. The data processing efficiency is improved.

This invention discloses a concurrency control and traffic scheduling system for card data interaction, relating to the fields of Internet of Things communications and distributed data processing technology. By adding a semantic addressing intent field (PIF) to the IPv6 extension header, the system enables switches to complete parsing and direct traffic to SmartNICs within 16ns. The SmartNIC generates a time-slot micro-ledger with a 500µs cycle and pre-allocates transaction locks using the Edge-Lock Weaving algorithm. Backend nodes complete strongly consistent transaction submissions using a single round of RDMA using a fast two-phase commit protocol. Dynamic congestion management is achieved through a dual-layer token bucket combined with flight windows. This solution achieves nanosecond read and write speeds, sub-millisecond submissions, and highly available decentralized scalability, significantly improving latency, throughput, and fault tolerance.