53 results about "Aggregation methods" patented technology

In order to extract more contextualized metrics associated with a particular service in a microservices-based application, a method of aggregating metrics comprises analyzing tags of spans for each trace and, based on a first predetermined set of rules, extracting one or more tags pre-selected by a user of the monitoring service. A pre-selected tag may appear in one or more spans associated with a particular service in a trace and may be extracted as a numeric metric. The method comprises computing a set of metrics for the particular service using the numeric metric extracted from the one or more spans of the trace. The method also comprises determining a set of aggregated metrics over the selected time duration for the particular service by aggregating the set of metrics over multiple traces.

The invention discloses a power system multi-source data intelligent fusion method based on a soft merging D-S evidence theory, and relates to the technical field of power system industrial multi-source heterogeneous data usage, comprising: acquiring multi-source data, and calculating a change index of the multi-source data; performing data detection on the multi-source data to obtain a detection result; performing aggregation processing on the multi-source data based on the change index and the detection result to obtain data features; and fusing the data features to obtain a data fusion result. According to the method, abnormal data is detected by aiming at the volatility and change characteristics of the power grid data and combining a Z-score method, an aggregation mode is flexibly selected, and the accuracy of data processing is improved; multi-source heterogeneous data are fused by using a D-S evidence theory, and in the face of evidence conflicts, a soft merging method is used for introducing regulatory factors to relieve conflict influences, so that the problem that time scales are not matched in the use process of industrial source data is effectively solved.

The invention belongs to the technical field of power grid dispatching, and particularly discloses a dual-port equivalent cluster aggregation method. According to the method, an independent aggregation operation interval is established for two ports by solving a specific robust optimization problem, an initial aggressive boundary is formed, and an initial high-dimensional polyhedral model is established. The key improvement is that the condition extreme value of the power of the other port is solved under the constraint of the operation interval of one port, so that the physical coupling relation is converted into a series of accurate boundary points, and meanwhile, the calculation complexity is also reduced. And finally, cutting the initial and aggressive high-dimensional model by using the boundaries, eliminating an infeasible space generated by neglecting coupling, and quickly generating a feasible dual-port aggregation interval. The problems of large model error and low aggregation efficiency caused by dual-port interconnection are systematically solved.

The invention relates to the technical field of transaction data aggregation, and discloses a near-real-time aggregation method and system for heterogeneous transaction system data, and the method comprises the steps: obtaining real-time change data from a plurality of heterogeneous transaction systems through a change data capturing technology, and standardizing the change data into a uniform format; performing Hash routing calculation on the standardized change data based on the order number to generate a Hash value, and routing the change data of the same order to the same message queue according to the Hash value; an independent consumer thread is allocated to each message queue through a multi-queue parallel processing mechanism, and the number of the consumer threads is dynamically adjusted according to the message accumulation amount; according to the method, the processed change data is aggregated according to the preset core field template, the aggregated data is stored in the aggregation database, cross-system unified query and analysis are supported, and the problems that heterogeneous data query is difficult, order affiliation judgment is complex, cross-system data statistical analysis is obstructed, and data synchronization timeliness is not high are solved.

The invention provides a clustering aggregation repetitive operation and maintenance order dispatching method, system and device and a medium, and belongs to the technical field of work order processing, and the method comprises the steps: obtaining a work order, and carrying out the preprocessing; extracting work order features from the preprocessed work order; unsupervised clustering is carried out on the feature vectors, the cosine similarity of any two feature vectors is calculated, and a neighborhood radius threshold value and the minimum point number are set; identifying a core point, and attributing all points in a neighborhood of the core point to an initial cluster where the core point is located; recursively checking the neighborhood of the points in the neighborhood, and finally forming a preliminary clustering result; clustering parameters are optimized according to the evaluation result, and a final clustering result is determined; common features of the feature vectors in the cluster are analyzed, and an interpretation text is generated; and merging the same cluster work order and sending the same cluster work order to a corresponding processing party. Through cooperation of a plurality of steps, full-process automation from original operation and maintenance work order data processing to efficient order sending is realized. The accuracy and efficiency of work order processing are effectively improved.

The invention discloses an intelligent aggregation method and engine for multi-source data, and relates to the technical field of data aggregation, and the method comprises the steps: firstly carrying out the segmentation and vectorization mapping of a multi-source data stream containing resumes, news and documents, retaining an original index, and analyzing a fine-grained atomic fact through a fine-tuning large model; then, cross-source alignment and clustering are carried out on the fact tuples, deep logic implication reasoning and conflict detection are executed on the basis, and the support and exclusion states among facts of different sources are quantified; furthermore, truth value optimization is completed by using a global consistency scoring mechanism, and a high-confidence core fact is locked from complex contradictory information. And finally, generating an abstract based on a preferred fact, and establishing an evidence traceability link from the generated content to the source data by using the original index. Thus, logic verification can be fused into the generation process, conflicts and model illusion of multi-source heterogeneous data in the space-time dimension are effectively eliminated, and logic self-consistent, true, credible and traceable intelligent aggregation is achieved.

The invention relates to the technical field of the Internet of Things, and discloses a method, a system and equipment for aggregating multi-dimensional data with dynamic dimension expansion in the Internet of Things, and a medium. According to the method, through an elastic modulus set and a pre-distributed redundant prime number, historical data and global parameters do not need to be reconstructed when dimensions are newly added, and optimized Paillier homomorphic encryption and CRT coding are fused, the calculation burden of the encryption process can be effectively reduced, and through batch verification at edge nodes, the communication and calculation overhead is remarkably reduced, and the communication efficiency is improved. A multi-layer data verification mechanism is provided by integrating Bulletprofs zero knowledge proof and a threshold Schnorr signature, false data injection, man-in-the-middle attack and data tampering can be effectively prevented, the integrity and authenticity of data in the aggregation process are ensured, the confidentiality of original data is ensured by using homomorphic encryption and CRT coding, and even if an aggregation ciphertext is intercepted, the security of the aggregation ciphertext is ensured. And an attacker cannot decrypt a single data value.

According to the load aggregation method and device based on the network load graph, by constructing the dynamically updated network load graph and establishing the network load coupling relation model, the constraint of a power grid topological structure on load adjustment is accurately quantified, and matching of a regulation and control strategy and a real-time system state is ensured; by introducing a multi-dimensional comprehensive aggregation evaluation system, the differences of the load in the aspects of characteristics, economy, response speed and the like are fully considered, and refined portraying and accurate aggregation of load resources are realized; and finally, through combination of multi-time scale aggregation and a multi-objective optimization algorithm, system operation indexes which conflict with each other are effectively coordinated, and an optimal load aggregation scheme which can quickly respond to system changes and can comprehensively optimize the regulation and control effect, the economic cost and the response speed is generated.

Systems and methods for partitioning forecast items into segments, or groups, based on an attribute. Once partitioned, segments are aggregated based on a predetermined memory size limit using an aggregation method. Aggregation methods include methods for providing segments that have a similar number of records. Segmentation is further enhanced by including variance reduction gain as a metric for selecting the order of attributes.

Embodiments of the present application provide a data aggregation method, a data aggregation device, an electronic device, a chip, a storage medium and a computer program product. The method comprises: performing a desensitization aggregation operation on target data in a target database according to aggregation query configuration information to obtain a desensitized aggregation data table; wherein the aggregation query configuration information comprises: a dimension field, a metric field, a desensitization rule and an aggregation rule; the dimension field and the metric field represent the target data, the desensitization rule represents a desensitization manner of a specific field, and the aggregation rule represents an aggregation manner of a specific field.

This invention relates to the field of database management technology, providing a streaming aggregation method, system, and device in a database. The method includes: establishing a mapping relationship in the database; responding to a user creating a streaming view containing aggregation operations, determining the corresponding final calculation function based on the mapping relationship, and creating a materialized table for storing intermediate aggregation states; rewriting the streaming view as a query that executes the final calculation function on the materialized table; responding to data being written to the streaming table, generating a first query plan based on the original query of the streaming view, generating a second query plan and a third query plan based on the mapping relationship, and executing them sequentially to first obtain the intermediate states of the current write operation, then merging the current write intermediate states with the existing intermediate states in the materialized table and updating the materialized table; responding to a user querying the streaming view, executing the rewritten streaming view, obtaining and returning the aggregation result. This invention can fully implement streaming aggregation functionality without modifying the database kernel.

The invention belongs to the technical field of artificial intelligence and data security, and relates to a secure aggregation method and system based on a reputation model and lightweight mask reconstruction. The method comprises the following steps that: 1) each client performs local model training locally, performs disturbance encryption on model parameters by using a lightweight mask mechanism after completing the local model training, and uploads the encrypted model parameters to an aggregation server; 2) the aggregation server descrambles the encrypted model parameters through a lightweight redundancy mask mechanism to obtain descrambled model parameters; 3) calculating the reputation of each client by using the reputation model based on the model updating validity and the model updating stability to obtain a comprehensive reputation value of each client; and 4) on the basis of the descrambled model parameters, carrying out weighted aggregation in combination with the comprehensive reputation value of each client to generate global model parameters, so that the security and robustness of model parameter aggregation in the federated learning process can be improved.

The application provides a power hierarchical routing aggregation method and system, which defines an aggregation window, configures a total timeout time, a tentative correlation parameter for the aggregation window, and initializes a loss count, a tentative probability and a waiting set of each sub-flow, starts the aggregation window to receive sub-flow data, and makes tentative judgment on the aggregation window according to the receiving result, sends a pre-warning packet to a missing sub-flow through the tentative result and the tentative probability, collects a pre-warning reply, updates the loss count, the tentative probability and the waiting set of the sub-flow through the pre-warning reply, waits for the sub-flow according to the update state of the waiting set, updates the loss count and the tentative probability of the sub-flow in the waiting set through the waiting result, and performs data packet aggregation forwarding after the waiting set is empty and the tentative initiation condition is not met or after the aggregation window times out, resets the aggregation window and performs the next round of aggregation after forwarding, so that the balance among aggregation efficiency, data integrity and resource saving is achieved through the power hierarchical routing aggregation method.

The invention provides a data-free federated knowledge distillation and aggregation method, which comprises the following steps of: processing a knowledge distillation and aggregation process of a model through a federated learning framework consisting of a server and a plurality of clients: when the clients participate in model training, training a local model by using a data set and a lightweight transfer model locally; the knowledge of the local model is fed back to the transfer model through a training process, the server executes global model aggregation and knowledge distillation of the transfer model, and the global model and a distillation result are returned to the client; in the knowledge distillation and aggregation process, the client and the server do not depend on any public data set; the client uses the transfer model to assist in training of the local model, and performs knowledge distillation on the transfer model to obtain model knowledge of other clients; and after the local model finishes a round of training, the server generates data for distillation by calculating the average of logic values output by different transfer models, and sends the data back to the client.

The application provides a traffic flow prediction method combining spatial multi-scale and time sequence multi-granularity structure, and belongs to the field of space-time data mining. First, the original data is preprocessed, and a suitable space-time aggregation method is selected to obtain multi-space scale data and multi-granularity time sequence data. Second, the multi-space scale data under the time sequence granularity of the original data is input into a multi-space feature mining module to learn and fuse the multi-scale space correlation between traffic units. Then, the multi-time sequence granularity data under the multi-space scale is input into a multi-time feature mining module to learn and fuse multi-granularity traffic time information. Finally, a multi-time and space feature fusion module is used to effectively aggregate the multi-space scale and multi-time traffic mode features, and output the final prediction result. Compared with the traditional method in the embodiment, the application has obvious advantages in prediction accuracy, can effectively improve the traffic flow prediction efficiency, and can realize efficient prediction.

The invention discloses a tensor clustering and aggregation method for dynamic equivalent modeling of a wind power plant. The method comprises the following steps: firstly, establishing a detailed model of the wind power plant and simulating a dynamic trajectory of the detailed model; selecting dynamic trajectory data of each wind driven generator state in the wind power plant to construct a tensor data set for describing the working conditions of the wind driven generators; an improved CP decomposition algorithm is adopted to carry out dimensionality reduction on the tensor data set, wherein the dimensionality reduction retains original structure information; clustering the wind driven generators on the dimension-reduced data set by using a k-means clustering algorithm; the wind driven generators belonging to the same category are equivalent to one aggregated wind driven generator; and calculating equivalent current collection system parameters to obtain a wind power plant equivalent model comprising the aggregated wind power generator and the equivalent current collection system. The tensor method is used for fan clustering, the high-dimensional characteristic of the dynamic trajectory of the wind power plant is considered, and the accuracy of the aggregation model is effectively improved.

The invention provides an NAS-based data verification, restoration and aggregation method and system, and the method comprises the steps: generating a real-time data flow according to the real-time access data and real-time shared data of each user in a network in the NAS, and carrying out the preliminary verification to obtain a plurality of vulnerability character strings of the real-time data flow, then deducing access behaviors / sharing behaviors of the vulnerability character strings, generating related behavior characteristics corresponding to each access behavior / sharing behavior according to the real-time data stream, performing character repair on the corresponding vulnerability character strings, and performing data segment division and classified storage; according to the method, the migration guide corresponding to the data source is constructed according to the migration track corresponding to each data segment, and the storage node corresponding to the suspicious data segment is adjusted according to the migration difference between the migration guides corresponding to each data source at different moments, so that the data security, the repair reliability and the access efficiency of the NAS system are improved.

The present application relates to a kind of Web map-oriented overlay aggregation method, belong to electronic map technical field.The present application will be divided into an aggregation layer for the zoom layer with similar aggregation effect, and set the aggregation layer level number according to the size of Web map zoom ratio.Based on the thought of exponential smoothing method, Web map aggregation layer prediction model is constructed, the screen coordinates of Web map container are calculated through browser window, and the geographic coordinates of map visual area are calculated using the model, then overlay data screening and aggregation calculation are carried out, finally the calculation result is cached with its corresponding aggregation layer level number.When map rendering, the cached data matched with current aggregation layer level number is used for aggregation rendering operation.The present application reduces the loading time of Web map in large data aggregation scenario, and improves the smoothness of browser.

The invention provides a federated learning robust aggregation method and system for resisting backdoor attacks, and belongs to the technical field of network and information security. The method comprises the following steps: acquiring model update vectors submitted by a plurality of clients, and decomposing the model update vectors into a plurality of hierarchical update data according to the hierarchical relationship of a global model; according to each hierarchy update data of each client, extracting a multi-dimensional feature of the client; determining a global total abnormal score of the client according to the multi-dimensional features of the client; and calculating a comprehensive reputation of the client according to the global total abnormal score, determining an aggregation weight according to the comprehensive reputation, aggregating updating data of each level of the client according to the aggregation weight, and updating the global model. According to the method, the attack success rate of malicious updating can be remarkably reduced, and the method has the advantages of better robustness and safety.

The present invention discloses a method, system and medium for judging the performance of API interfaces based on link tracing. The method includes: each agent accesses the corresponding distributed service without code intrusion, actively scans all Web API interfaces in the distributed service when the distributed service is started, obtains the full amount of static data of the Web API interfaces and reports it to the center; obtains the HTTP requests of external services, realizes link tracing through each agent, and obtains the link tracing data of each HTTP request reported by the agent to the center at regular intervals; calculates the average performance indicators of each Web API interface by using the interface-based aggregation method; the center calculates the interface behavior characteristic data according to the link tracing data; the center uses a comprehensive judgment model to conduct a comprehensive evaluation based on the full amount of static data, average performance indicators and interface behavior characteristic data, and performs timely response processing on the interfaces corresponding to the key attention scores.

The application relates to a wind farm dynamic equivalent aggregation method based on PCA and Kernel-FCM and belongs to the technical field of power system control. Specifically, the method comprises the following steps: obtaining the unit dynamic characteristics in a preset wind turbine mathematical model, selecting multiple clustering indexes according to the unit dynamic characteristics, calculating the initial values of state variables corresponding to the clustering indexes, constructing high-dimensional data samples from the initial values of the state variables, performing dimension reduction processing on the high-dimensional data samples based on a PCA analysis method to obtain low-dimensional data samples, determining the optimal clustering number by balancing simulation accuracy and calculation amount based on a Gap Value, performing cluster division on the low-dimensional data samples based on a Kernel-FCM algorithm, calculating dynamic equivalent parameters of a power collection network based on a fast analytical method, and constructing a wind farm dynamic equivalent aggregation model based on the equivalent unit parameters and the dynamic equivalent parameters of the power collection network. The application can improve the reliability and accuracy of clustering results.

The application provides a multi-constraint-based spatial element dynamic aggregation method and system, and belongs to the technical field of spatial data processing. The method comprises the following steps: acquiring shared bicycle parking point spatial element data set and multi-dimensional constraint conditions; performing spatial profiling on a target research area based on spatial division constraints, generating a plurality of initial aggregation service range units, and mapping elements in the spatial element data set into the initial aggregation service range units, and calculating statistical indexes of the initial aggregation service range units; constructing a graph structure with the initial aggregation service range units as nodes, and the spatial adjacency relationship and semantic similarity between the units as edges; constructing a graph structure aggregation model, including a target function with the minimum variance of the statistical indexes between the initial aggregation service range units as the target, and a constraint condition based on the integrity of the spatial division constraints; and solving the graph structure aggregation model by using a heuristic search algorithm or a graph partitioning algorithm to generate a dynamic aggregation service range result satisfying the statistical balance constraint.

One or more embodiments of this specification provide an aggregate query method, system, device, and storage medium. The method includes: responding to a database aggregate query request, determining the data table to be queried, the grouping field, and the aggregation method; creating a grouping array; dividing the data table into at least two data sub-tables, and sequentially processing each data sub-table as follows: for multiple tuples in the data sub-table, performing the following operations in parallel: checking whether there is a target grouping field in the grouping array that is the same as the grouping field of the tuple; if so, recording the array index; after traversing all tuples in the data sub-table, for tuples to be processed that do not have an array index recorded, adding the grouping field of the tuple to be processed to the grouping array and recording the array index; performing aggregate calculations based on the array index, grouping array, and aggregation method recorded for each tuple to obtain an aggregate query result. This is conducive to improving the efficiency of aggregate queries.

This invention relates to the field of e-commerce data processing technology, specifically to a method and apparatus for multi-platform e-commerce order aggregation based on dynamic clustering rules. The technical solution includes: collecting order data from e-commerce platforms and performing standardized processing to obtain platform order data; extracting various clustering feature values ​​from a clustering feature system; matching these values ​​with rules in a dynamic clustering rule set to determine the candidate cluster to which the order belongs; calculating the comprehensive similarity between orders; determining whether the comprehensive similarity is greater than or equal to a preset similarity threshold, and assigning the order to that cluster; performing missed order detection for orders not clustered; and outputting the order results for the clusters. This significantly improves clustering accuracy, effectively reduces missed orders, and minimizes operating and transportation costs. It also significantly improves order processing efficiency, adapting to the needs of large-scale order processing. The system operates stably and supports efficient processing of massive amounts of data.

The present invention relates to the technical field of federated learning, and specifically provides a federated aggregation method and device for data imbalance, which has the following steps: S1. Construct a data quality vector, which consists of a gradient factor, a distribution factor, and a quantity factor; S2. Use the data quality vector as a clustering feature to perform clustering analysis on the participants to achieve participant grouping; S3. Complete the global gradient calculation based on the method of aggregating gradients by group. Compared with the prior art, the present invention constructs a data quality description vector, weighs the quantity, quality, and model contribution of the data of each participant in the case of data imbalance, measures the differences between data sets from multiple angles, and the clustering analysis based on this vector can greatly improve the communication efficiency.

The invention provides a federated distributed aggregation method and device, electronic equipment and a storage medium, the method is applied to the technical field of machine learning, and the method comprises the following steps: segmenting a global secret s in a target global model training process into n secret segments, and sending the n secret segments to n model training devices; for any model training device, determining a secret key of the model training device and sending the secret key to the model training device, so that the model training device encrypts the original model parameters to obtain encrypted model parameters; segmenting the encryption model parameters to obtain m encryption shares; aggregating the m encrypted shares to obtain an aggregated share of the model training device; and updating the target global model based on the aggregated share of the n model training devices. According to the method, the data in the training process can be encrypted in the model training process of the federated distributed method, so that the privacy and security in the model training process are improved, and the success rate of model training is ensured.

The invention provides a federated learning robust aggregation method and system fusing layer-by-layer dynamic clustering and statistic adaptive preferential selection, and belongs to the technical field of federated learning. The method can dynamically cluster and screen benign update layer by layer: after a server receives client model update, the benign update is accurately identified through a hierarchical clustering mechanism, a randomized clustering algorithm selection strategy is introduced, each layer of clustering algorithm is dynamically determined, and clustering robustness is enhanced; based on a hierarchical clustering result, a statistical magnitude calculation mode (such as a mean value and a median) is selected through a random number, so that random statistical magnitude self-adaptive preferential selection is realized; the situation that a single statistic is dominated by an abnormal value is avoided, the influence of abnormal data on global model updating is reduced, multi-source information is fused to generate global model updating, abnormal value interference is effectively restrained, and the adaptability to complex data distribution is improved.

The invention belongs to the intelligent power grid industry, particularly relates to the technical field of intelligent scheduling systems, and discloses an equivalent cluster dual robust aggregation method considering source load uncertainty. The method comprises the following steps: firstly, establishing an equivalent cluster model containing uncertainty; the feasibility of the scheduling scheme under the worst uncertainty is ensured through the first min-max robust optimization; secondly, the worst influence of uncertainty on instruction execution is actively evaluated through second max-min robust optimization, the scheduling instruction tends to be conservative by maximizing the correction cost, and therefore the aggregation interval is actively reduced; and finally, combining the results of the two reconstruction modes to carry out iterative solution until convergence. According to the method, the finally generated aggregation operation interval is more compact and reliable while the uncertainty influence is fully considered, it is ensured that any scheduling instruction in the interval can be safely executed when facing any uncertainty, and the problem that the interval reliability is poor is fundamentally solved.

This invention provides a data verification, repair, and aggregation method and system based on NAS, comprising: generating a real-time data stream in the NAS based on each user's real-time access data and real-time shared data in the network, performing preliminary verification to obtain several vulnerability strings in the real-time data stream, then deriving the access / sharing behavior of the vulnerability strings, generating relevant behavioral characteristics corresponding to each access / sharing behavior based on the real-time data stream, performing character repair on the corresponding vulnerability strings and dividing and classifying the data segments for storage, constructing migration guidelines for the corresponding data source based on the migration trajectory corresponding to each data segment, and adjusting the storage nodes of the corresponding suspicious data segments based on the migration differences between the migration guidelines corresponding to each data source at different times, thereby improving the data security, repair reliability, and access efficiency of the NAS system.

A method for processing aggregated records. Receive an aggregated record. Identify key-value pairs within the aggregated record. The key-value pairs are composed of a key and a value, and the key is composed of a deduplication marker. Identify an aggregation method based on the characteristics of the deduplication marker. Process the aggregated record using an algorithm determined by the aggregation method to generate at least one processed record.