42 results about "Merge sort" patented technology

The invention relates to the technical field of electrical digital data processing, and discloses a statistical entropy and multi-factor fused text similarity data processing method, which comprises the following steps that: a processor extracts substring sets which do not contain maximum common values of a first data sequence and a second data sequence, and calculates the quadratic sum of the lengths of substrings to generate local statistical entropy; traversing the maximum common substring set to obtain storage address indexes of the maximum common substring set in the first data sequence memory space and the second data sequence memory space, and constructing a topological mapping vector of a mapping structure displacement relationship; calculating the total number of inverted pairs of the topology mapping vector by using a merge sorting algorithm, and generating a normalized topology dissipation index; and by taking the local statistical entropy as an information carrier and taking the topological dissipation index as a structural damping factor, executing nonlinear damping modulation operation to obtain a final similarity score, and solving the technical problem that the block-level displacement cannot be identified by linear scanning logic by quantizing topological entropy increase of data distributed in a storage space.

The invention discloses a production line data integration method based on a digital twin model, and belongs to the technical field of electronic data processing. The production line data integration method comprises the following steps: step S0, pre-preparation and standard definition; the method comprises the following steps: S1, data acquisition and preprocessing; step S2, data cleaning and integration; s3, data analysis and mining; 4, constructing a digital twinborn model; and 5, managing system-level data. The method has the following advantages: data islands are cracked, data quality and processing efficiency are improved, internal business logic of data is mined, high-precision feature extraction and pattern recognition are realized, and the method is suitable for popularization and application by means of unifying data standards and cleaning rules, optimizing a merge sorting strategy, building a double-layer association analysis model, adopting a CNN-LSTM hybrid model, perfecting data security and authority management and the like. The method supports the cross-line collaborative decision and precise production optimization, finally solves the problem that the prior art cannot support the system-level digital twinning application, and improves the application efficiency and effect of the digital twinning system.

The invention discloses a data processing device, a data processing method, a chip and electronic equipment, and relates to the technical field of data processing. The data processing device comprises a storage module which is configured to store x groups of original data vectors, the original data vectors comprise m data elements, and x and m are positive integers; the calculation module is configured to access the original data vectors stored in the storage module, and reconstruct the x groups of original data vectors into one or more intermediate matrixes according to the parallel processing capability of the single-instruction multi-data execution unit and a target value k of a Top-k operator; loading the intermediate matrix into a vector register block according to columns, carrying out odd-even merging sorting by utilizing a single-instruction multi-data execution unit, and realizing in-line sorting on the intermediate matrix; and loading the intermediate matrix subjected to in-line sorting into a vector register block according to columns, and performing one or more rounds of merging sorting by utilizing a single-instruction multi-data execution unit to obtain top-k data elements in x groups of original data vectors.

A method and apparatus for performing single-stage merge sorting of at least two pre-sorted input lists. The method and apparatus can provide a stable merge such that a value in an input list with a higher stable property will precede an equal value in another input list with a lower stable property when mapped into the single sorted output list. The method and apparatus preferably perform the merge sorting in a set of single-stage operations and are most preferably implemented in asynchronous combinatorial logic, which uses no internal clocking—although synchronously clocked operations can be used and will provide desirable results.

Provided is a data compaction method that includes: determining, by an electronic device, a plurality of Sorted String Table (SSTable) files on which a compaction operation is to be performed among SSTable files stored in a storage device connected to the electronic device, the data in the SSTable files stored in the storage device is stored based on a Log Structured Merge Tree (LSM-Tree) mechanism; based on receiving the plurality of SSTable files from the storage device, performing, by the electronic device, merge sorting on data in data blocks of the plurality of SSTable files based on indexes of the data in the data blocks of the plurality of SSTable files, to obtain reconstructed data blocks; and sending, by the electronic device, the reconstructed data blocks to the storage device, the reconstructed data blocks being used by the storage device to obtain reconstructed SSTable files.

The invention discloses a distributed database incremental snapshot method and device and computer equipment. The method comprises the steps that a change log from a distributed database node is obtained, and the change log comprises transaction key information; performing transaction assembly on the change log to obtain an assembly result; using a starting timestamp of the transaction as a sorting key, and merging and sorting the assembly results through a log structure merging tree structure to generate an ordered lake format file; generating a snapshot ID reflecting a current database state by using a Paimon interface based on a transaction starting timestamp or a time field in a table; and querying the ordered lake format file and the snapshot ID by using a standard data lake reader. By implementing the method provided by the invention, the data migration process from the distributed database to the data lake can be simplified, the efficiency and reliability of the whole process are improved, and the consistency and integrity of the data during transmission among different systems can be ensured.

The invention provides a regular batch data processing method and system and a storage medium, and relates to the technical field of batch data processing. The method comprises the following steps: importing collected field data and storing the field data in a database, creating a data index for the field data and storing the data index in a cache; sorting the data indexes in the cache by using an improved merge sorting algorithm to obtain new data indexes; setting a timed task, acquiring field data in batches from the database according to the new data index and the capture frequency of the timed task, and determining a target data processing thread through a distributed lock mechanism to acquire the field data; and forwarding the obtained field data, and sending a processing result to a specified terminal and displaying the processing result. According to the method, the data is captured through the cache according to the sorted index, the response time and the database access frequency are reduced, the field data are processed in real time in the message middleware through the timed task, a visual visual interface is generated, and the data processing and interaction efficiency is improved.

The invention discloses a pulse description word sorting method based on arrival time, which comprises the following steps of: selecting PDWs of which existing TOAs are not sorted from small to large from an original PDW sequence, and storing the PDWs. And sorting the stored PDWs from small to large according to the TOA values. And finally, according to the TOA values from small to large, merging and sorting the sorted stored PDWs and the original remaining PDWs to form a new PDW sequence. According to the method, the PDWs of which the TOA values are not ranked from small to large can be quickly selected from the original PDW sequence, and the PDWs can be quickly merged and ranked to form a new PDW sequence, so that conditions are provided for the next PDW data playback or signal sorting, the data playback or signal sorting can be smoothly carried out, and the success rate of the data playback or signal sorting is improved. And meanwhile, the phenomenon that the PDW data is interrupted or cannot be played back when being played back is avoided.

The invention discloses a zero-knowledge proof-oriented multi-scalar multiplication hardware parallel acceleration method and device, and relates to the technical field of cryptographic acceleration and parallel computing. The method is based on a field programmable gate array, an improved multi-channel parallel Pippenger algorithm is utilized, and the parallel acceleration of the multi-scalar multiplication hardware is realized. The method comprises the following steps: firstly, performing scalar preordering, performing conflict-free data scheduling and designing a compact large integer multiplier to accelerate large-scale elliptic curve scalar multiplication, and selecting a window size and a precomputation scale by adopting an adaptive extensible 3D-Pippenger algorithm; in a data stream level, conflict between empty bucket reading and bucket writing is avoided through scalar slice merging sorting and pointer caching strategies. The acceleration circuit is suitable for various elliptic curves and different zero-knowledge proof protocols, and can be widely applied to scenes such as block chain private transactions, verifiable database outsourcing and verifiable machine learning.

The invention discloses a multi-energy and carbon emission correlation analysis method and system for a high-energy-consumption enterprise, and relates to the technical field of energy management and carbon emission accounting, and the method comprises the steps: carrying out the preprocessing of historical multi-source data of a high-energy-consumption target enterprise, and constructing a training set and a test set which are distributed in a balanced manner; then introducing a shuffled frog-leaping optimization algorithm to perform global optimization on the network structure and parameters of the deep extreme learning machine, determining the optimal hidden layer structure and weight configuration of the model through an iteration mechanism of subgroup division, local jump and merge sorting, and constructing a multi-energy consumption prediction model with efficient deep feature extraction capability; a coupling feature matrix generated based on real-time multi-source data is input into a multi-energy consumption prediction model to obtain a multi-energy consumption prediction result, finally, association analysis is performed by combining real-time data, the prediction result and an energy carbon emission coefficient, the association strength of multi-energy and carbon emission is quantified, and the energy consumption prediction accuracy is improved. And more accurate and reliable data support is provided for energy structure optimization and emission reduction decision making of high-energy-consumption enterprises.

The invention discloses a robot cluster optical network communication method and system based on optical communication data, and relates to the field of robot communication, and the method comprises the steps: carrying out the logic partitioning of a robot based on a coupling communication link and a partitioning instruction of a central control module; the robots establish an optical communication network between the robots in each logic partition according to the logic partitions and an autonomous posture adjustment positioning mechanism; carrying out local ordering on each logic partition based on a partition two-way bubble merging sorting algorithm; determining a minimum robot, a maximum robot and boundary information of each logic partition according to a fuzzy immune clone selection algorithm; data merging and boundary ordering are carried out between the logic partitions; and performing global broadcasting according to the distributed negotiation model. According to the method, the problems of low efficiency, wireless communication bottleneck and consistency of existing distributed sorting are effectively solved through an integrated process of robot partition efficient sorting, boundary collaborative merging, distributed broadcast negotiation and the like.

This invention relates to the field of artificial intelligence technology, providing a data sorting method, electronic device, storage medium, and computer program product. The method includes: performing local sorting of data to be processed in parallel using multiple processing units of a parallel processor; each processing unit acquiring a portion of the data to be processed; sorting the acquired portion of data and selecting a preset number of data points as local target data; and performing one or more merge sort operations on the local target data acquired by each processing unit, based on the amount of data to be processed and the hardware resources of the parallel processor, and selecting a preset number of data points as global target data from the merged and sorted data. This invention avoids the huge computational overhead of globally sorting all data to be processed, significantly improving the efficiency and performance of TopK operations.

According to the multi-party data security grouping method and device provided by the embodiment of the invention, when multiple pieces of target data in a target data set commonly and securely held by multiple participants are grouped according to a predetermined attribute, each participant can firstly disorder the target data set; the method comprises the following steps of: selecting target data from a plurality of participants to cut off the control of each participant on the position information of the target data, then sorting out-of-order target data sets according to attribute values on predetermined attributes through quick sorting and merging sorting based on security comparison to obtain ordered data sets, and then performing data grouping based on the ordered data sets by using the security comparison. According to the technical scheme, the security grouping efficiency can be improved.

This application relates to a hardware-implemented sorting method, apparatus, computer device, readable storage medium, and program product. The method includes: acquiring a first data sequence to be sorted, where each data element in the first data sequence contains a key value and its position index within the sequence; concatenating the key value and position index of each data element according to a preset sorting direction to obtain the first data elements to be sorted; and performing hardware parallel sorting on the first data elements using a bitonic merge sort network to obtain a sorted second data sequence, wherein the sorting order of the data elements in the second data sequence retains the relative positional relationships in the first data sequence. This allows the primary sorting order to be determined based on the key value, and the relative order of data with the same key value to be determined based on the positional information. While maintaining sorting stability, it inherits the hardware parallel advantages of bitonic merge sort, resulting in fast sorting speed suitable for real-time video decoding requirements.

The present invention belongs to the field of information security technology, and discloses a numerical security sorting method and system based on ciphertext, wherein the numerical security sorting method includes: a numerical security sorting algorithm based on ciphertext, using a distributed encryption key to encrypt the private data held by multiple parties; sending the ciphertext data to a cloud server, and the cloud server uses a data size comparison protocol to compare the ciphertext data, and then combines a merge sorting algorithm to sort the ciphertext data. The present invention constructs a data comparison protocol based on a private key splitting method to ensure the privacy of the data, and uses a merge sorting mechanism to achieve high efficiency in ciphertext sorting. At the same time, in response to data security issues in distributed scenarios, the present invention provides an efficient privacy-protected numerical security sorting algorithm designed for a secure multi-party sorting model, which can ensure the security of sensitive information in the multi-source data sorting process, and effectively promote and drive the vigorous development of big data related technologies.

The application discloses a distributed database incremental snapshot method and device and computer equipment. The method comprises the following steps: obtaining a change log from a distributed database node, wherein the change log comprises transaction key information; assembling the change log to obtain an assembly result; using a transaction start timestamp as a sorting key, performing merge sorting on the assembly result through a log structure merge tree structure to generate an ordered lake format file; using a Paimon interface to generate a snapshot ID reflecting a current database state based on a transaction start timestamp or a table time field; and querying the ordered lake format file and the snapshot ID by using a standard data lake reader. The method of the application can not only simplify the data migration process from the distributed database to the data lake, but also improve the efficiency and reliability of the whole process, and ensure the consistency and integrity of the data transmission between different systems.

The present invention provides a method, system and medium for fast collection and filtering of distributed link error and slow logs, including: extracting log data within a specified time range; storing the log data in a circular buffer, with the write pointer pointing to the buffer block; judging each log data item one by one to determine if the status is not 200, and recording it as a log with a status other than 200; storing the field traceId of the log with a status other than 200 in the error and slow log traceId set S1; reading the log data in the block pointed to by the read pointer in the circular buffer and the log data in the blocks before and after the pointer, and reading the problem traceId data D1 of the set S1; if the traceIds of the data in S1 and D1 match, it is marked as problem link log data; taking out the problem link log data and storing it in the error and slow log set L1; using the merge sort method to sort the traceId field id in ascending order for the data in the error and slow log set L1 to obtain the set R1; using the merge sort method to sort the time field time from earliest to latest for the set R1 to obtain the set R2; reporting the error and slow log links in the set R2.

The embodiment of the invention discloses a switching control method, a chip and an energy storage system, and the method comprises the steps: sorting state parameter groups of all energy storage modules in the energy storage system, and obtaining a plurality of first state parameter groups after in-group sorting; performing at least one level of merging sorting on each first state parameter group after sorting in the group to obtain each state parameter after merging sorting; determining a to-be-switched-out energy storage module and a to-be-input energy storage module in the current control period from the energy storage modules based on the merged and sorted state parameters; and based on the to-be-switched-out energy storage module and the to-be-input energy storage module, performing switching control on each energy storage module.

The present invention relates to a sliding window-based LSM data merge sorting method and device. The method proposed by the present invention first constructs a sliding window for a sorted string table at a low level in a merge operation, selects a low-level sorted string table participating in the merge operation based on the sliding window, and simultaneously obtains the two sorted string tables closest to the left and right of the sliding window, and obtains their key control ranges, and then selects a sorted string table in a high-level sorted string table that has an intersection with the key control range of the low-level sorted string table and obtains its key control range, and determines whether to perform a merge sort operation by estimating the key control range of the sorted string table after the merge, and comparing it with the key control ranges of the two sorted string tables closest to the left and right of the sliding window. The present invention reduces the number of times a high-level sorted string table repeatedly participates in the merge process of a low-level sorted string table, speeds up the merge sorting rate, and improves the system read and write performance.

The application relates to a data sorting method and device, a computer device, a storage medium and a computer program product. The method comprises the following steps: obtaining an initial data sequence, dividing the initial data sequence to obtain a plurality of basic sub-data sequences; taking each basic sub-data sequence as an initial sub-data sequence, combining each initial sub-data sequence to obtain a plurality of initial sub-data sequence combinations; performing merge sorting on each initial sub-data sequence in the same initial sub-data sequence combination to obtain an intermediate sub-data sequence combination corresponding to each initial sub-data sequence combination; taking the intermediate sub-data sequence combination as an initial sub-data sequence, returning to the step of combining each initial sub-data sequence until a termination condition is met, and obtaining an ordered data sequence corresponding to the initial data sequence. The method can improve the data sorting efficiency and can be applied to a multidimensional database.

The invention discloses a dialogue content generation method and electronic equipment, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a target vector which is output by an activation function of a dialogue model created based on a pre-training model and is constructed based on the sampling probability of each lexical element in a word list, carrying out the partitioning of the target vector, and obtaining a plurality of data blocks, and then, based on a target sampling probability threshold, screening out the first lexical units in each data block in parallel to construct corresponding target sets, then, merging and sorting the target sets to obtain a target list, and generating and outputting corresponding target dialogue content based on the target list. According to the method, the target vector output by the activation function is partitioned, then the lexical units in the data blocks are screened out in parallel to construct the corresponding target set, and then the target set is merged and sorted to obtain the target list to generate the corresponding target dialogue content, so that the hardware utilization rate is effectively improved, the data granularity of single processing is reduced, and the dialogue generation efficiency is improved.

The application provides a dynamic table merging method and device, equipment and storage medium, relates to the technical field of data processing, and the merging method comprises the following steps: obtaining an original list and a merging condition, and determining label data and non-label data in the original list according to the merging condition; wherein the original list comprises at least two original data groups, the original data group comprises a plurality of original data, the original data in the same column corresponds to the same data node; according to the label data and the non-label data, determining to-be-merged data corresponding to the label data and non-merged data corresponding to the non-label data; performing merging sorting on the to-be-merged data corresponding to the label data of the same type to generate merged data; and generating a target list according to the label data, the non-label data, the merged data, the non-merged data and the original list. The application performs merging processing on complex list data, reduces the calculation difficulty of the number of cross cells, and greatly reduces the merging calculation difficulty.

The present invention provides a post-processing method, device, and system for small target detection in large images. The method includes: looping and executing the following steps in parallel until all feature map arrays are traversed: S1, reading the feature map array obtained by NPU inference; S2, sorting the feature maps in the feature map array according to confidence and category; wherein, during sorting, a non-recursive merge sorting dedicated circuit structure is adopted, and sorting is performed in parallel for each merge step; S3, calculating the intersection-over-union ratio using a dedicated circuit structure, and filtering the sorted feature map array based on the intersection-over-union ratio to obtain candidate targets that meet the requirements; S4, transmitting the filtered candidate targets to the NPU. This embodiment uses a multi-stage pipeline method to implement the parallel expansion of the candidate box sorting algorithm in the feature map, thereby optimizing the timing relationship of the loop, minimizing the loop time, and improving the post-processing speed.

The invention relates to the technical field of database query and distributed systems, and discloses a performance optimization method and system for cross-data-source paging query, and the method comprises the steps: receiving a query request, and retrieving a historical deviation calibration factor; sending a statistical probe to the heterogeneous data source, aggregating the statistical probe into a global distribution estimation function, and calculating a physical value domain anchor point; constructing a differential locking window, and distributing a prefix counting instruction and a window data acquisition instruction in parallel; calculating a global reference displacement according to a returned counting result, merging and sorting window records, and intercepting a target result set; and calculating the error between the real ranking and the estimated ranking of the physical anchor points, and updating the historical deviation calibration factor according to the error. According to the method, logic offset is mapped into physical anchor points, index counting and local slicing are used for replacing linear scanning, and a deviation feedback self-learning mechanism is combined, so that IO overhead of cross-source deep paging query is remarkably reduced, and continuous optimization of query performance and positioning precision is realized.

The application discloses a kind of robot cluster optical network communication method and system based on optical communication data, it is related to robot communication field, this method includes: based on the partition instruction of coupling communication link, central control module, robot is logically partitioned;Robot establishes the optical communication network between each logical partition inside robot according to logical partition and self attitude adjustment positioning mechanism;Based on the local ordering of each logical partition by partition bidirectional bubble merge sorting algorithm;According to fuzzy immune clone selection algorithm, determine the minimum robot and maximum robot of each logical partition, boundary information;Data merging and boundary ordering are carried out between each logical partition;According to distributed negotiation model, global broadcast is carried out.The application solves the problems of low efficiency of existing distributed sorting, wireless communication bottleneck and consistency by integrated process of robot partition efficient sorting, boundary collaborative merging, distributed broadcast negotiation and the like.

The invention relates to the technical field of artificial intelligence, and provides a data sorting method, electronic equipment, a storage medium and a computer program product.The method comprises the steps that local sorting is conducted on to-be-processed data in parallel through a plurality of processing units of a parallel processor, each processing unit in the plurality of processing units acquires partial data from the to-be-processed data, and each processing unit sorts the acquired partial data and selects a preset number of data as local target data; and according to the data volume of the to-be-processed data and the hardware resources of the parallel processor, performing one or more merging and sorting operations on the local target data obtained by each processing unit, and selecting a preset quantity of data from the merged and sorted data as global target data. According to the method, huge calculation overhead caused by global sorting of all the to-be-processed data is avoided, and the efficiency and performance of TopK operation are remarkably improved.

The present invention relates to the technical field of data synchronization of heterogeneous data sources, and provides a multi-source data merging incremental extraction method and system, the method comprising: inserting all data records in a source table into a target table; respectively calculating the characteristic values ​​of the source table and the target table row by row; merging and sorting the characteristic values ​​in the calculated source table and the target table to obtain a first set and a first time complexity and a second set and a second time complexity; calculating a difference set; confirming the insertion record and the deletion record according to the difference set calculation result; respectively performing a sharding operation on the insertion data and the deletion data corresponding to the confirmed insertion record and the deletion record to form a plurality of insertion shards and a plurality of deletion shards; after the sharding operation, first executing a thread pool to batch delete each deletion shard, and then executing a thread pool to batch insert each insertion shard. The present invention is applicable to all relational database tables, and the database table does not need to have special fields such as auto-increment primary keys or timestamps.

The application discloses a dialogue content generation method and an electronic device, and relates to the technical field of artificial intelligence, and comprises the following steps: obtaining a target vector constructed based on sampling probabilities of each word element in a word table and output by an activation function of a dialogue model created based on a pre-training model; obtaining a plurality of data blocks by blocking the target vector; filtering out first word elements in each data block based on a target sampling probability threshold to construct a corresponding target set in parallel; performing merge sorting on each target set to obtain a target list; and generating and outputting corresponding target dialogue content based on the target list. By blocking the target vector output by the activation function, filtering out the word elements in each data block to construct the corresponding target set in parallel, and then performing merge sorting to obtain the target list to generate the corresponding target dialogue content, the hardware utilization rate is effectively improved, the data granularity of single processing is reduced, and the dialogue generation efficiency is improved.

A data processing method and system for a distributed database, and a device and a storage medium. The method comprises: sorting transactions in a transQueue by using a global timestamp TSO, creating a local transaction list by means of an XA Event and a waitTrans in the transQueue, and generating first-level sorting queues according to the local transaction list (102); for the first-level sorting queue of each DN, performing multi-path merging and sorting, so as to generate a global sorting queue (104); and for the global sorting queue, merging local transactions that have the same global timestamp TSO, and generating a global binary log Binlog (106).