22 results about "Block group" patented technology

The invention provides a privacy record linking method for low-quality data, which can be applied to the technical field of data fusion entity alignment. The method comprises the following steps: a link participant carries out multiple rounds of negotiation communication based on a link scene, and determines parameter information for subsequently executing a privacy record link; bloom filter coding operation is executed on respective data of link participants, the data missing number is recorded synchronously, and a coding table and a missing table are generated; the link unit analyzes the data of the coding table and the missing table by using the dynamic locality sensitive hash parameter, generates a record partitioning flag bit, and completes record partitioning according to the flag bit to form a partitioning group; a link unit extracts records in the corresponding blocks from the block group, marks the records in the blocks to form record pairs, and calculates a dynamic threshold value and Jaccard similarity of each record pair according to a reference filtering threshold value and a threshold value reduction amount; and in response to the Jaccard similarity being greater than or equal to the dynamic threshold, outputting the corresponding record pair identifier as a result.

The invention discloses a check disk data recovery method and device, electronic equipment and a computer readable storage medium. The method comprises the steps that a check disk corresponding to a diagonal check chain in a target redundant array is determined; for each verification block in the verification disk, grouping all target data blocks corresponding to the verification block to obtain each data block group; wherein all the target data blocks in each data block group belong to the same data disk; for each data block group, calculating all target data blocks in the data block group by using a data disk corresponding to the data block group to obtain an initial check value corresponding to the data block group; sending the initial verification value corresponding to each data block group to a verification disk, and calculating each initial verification value by using the verification disk to obtain a verification value corresponding to the verification block; and obtaining recovery data of the check disk based on the check values corresponding to all the check blocks, and completing data recovery of the check disk. According to the technical scheme, the data recovery of the check disk can be realized more quickly and efficiently.

A data storage method, and a storage system, a storage apparatus and a storage device, which are used for improving the utilization rate of a storage space of the storage system. The data storage method comprises: dividing each primary fault domain into a plurality of secondary fault domains, wherein each secondary fault domain comprises some storage apparatuses among a plurality of storage apparatuses, and the storage apparatuses comprised in each secondary fault domain are different from those comprised in one another; dividing data to be written into a plurality of data block groups, wherein each data block group comprises N data blocks, and N is an integer greater than 0; generating at least one primary parity block on the basis of the N data blocks in each data block group, so as to obtain a plurality of groups of primary parity blocks; and respectively storing, in storage apparatuses included in different second fault domains, different data blocks in each data block group, and then generating secondary parity blocks on the basis of the plurality of groups of primary parity blocks, wherein the secondary parity blocks are parity blocks for a plurality of groups of data blocks.

The invention discloses a full-to-full communication method for a multi-dimensional full-connection network topology, and belongs to the field of full-to-full communication, and the method comprises the steps: constructing the multi-dimensional full-connection network topology; determining the total number of processes participating in the Allall communication task according to the cardinal number and the dimension number; equally dividing a plurality of parts of data, participating in the Allall communication task, of each process into a plurality of data blocks according to dimensions; numbering and grouping each data block of each process based on the source process, the target process and the dimension of the data block to obtain a plurality of data block groups; each process communicates with an adjacent process of each dimension at the same time, and exchanges data blocks in the data block groups of different dimensions; repeating the exchange of the data blocks until the data blocks of all dimension groups are exchanged between each process and the adjacent process of each dimension; and recombining the data blocks into final complete data by each process according to the serial numbers of the data blocks. According to the invention, the problems of link congestion and low hardware utilization rate in the multi-dimensional fully-connected network topology are solved.

The present invention relates to the technical field of FPGA bitstream compression, and a method and system for fast compression of FPGA bitstreams, including: traversing line numbers in a target multi-layer index structure, identifying a data block group, a feature value group, and the number of data blocks corresponding to the line numbers, determining whether the number of data blocks is a preset number, if it is the preset number, then determining whether there is a subsequent single data block group set, if there is, then merging the data block group with the subsequent single data block group set to obtain a merged data block group, if not, then using the data block group as the original data block group, if it is not the preset number, then determining whether there are the same feature values in the feature value group, if there are, then compressing the data block group to obtain a compressed data block group, if not, then using the data block group as the original data block group, and performing bitstream recombination on the merged data block group set, the compressed data block group set, and the original data block group set. The present invention can solve the problems of poor overall compression effect of FPGA bitstream data and relatively large consumption of computing resources.

The application discloses a kind of data block group migration mapping table construction method, device, equipment and storage medium.The method comprises: according to the migration instruction of data block group, the data block copy identifier of each data block in data block group is obtained;Judge whether the target data block copy identifier of the current processing data block exists in copy identifier mapping table, if exists, from copy identifier mapping table, obtain reference snapshot information;Otherwise, according to the hit result of target data block copy identifier and the snapshot attribute of each snapshot in snapshot center, reference snapshot information is obtained;According to the reference snapshot information matched with each data block copy identifier, generate the data block group migration mapping table based on snapshot index.The technical scheme of the embodiment of the application, by establishing and using copy identifier mapping table, the reference result of previously migrated data block is reused, the construction efficiency of current data block group migration mapping table is improved, so as to improve the efficiency of data migration.

The application relates to the computer technical field and discloses a data processing method based on erasure code and related devices. The application obtains a plurality of original data blocks; divides the plurality of original data blocks into two sequential groups and a plurality of cross-group data block groups; the original data block serial numbers in the sequential groups are continuous, each cross-group data block group contains the same number of original data blocks, and the cross-group data block serial numbers are discontinuous; a plurality of check blocks of the plurality of original data blocks are generated, the plurality of check blocks include two global check blocks, sequential group check blocks corresponding to each sequential group, and cross-group check blocks corresponding to each cross-group data block group. Thus, in the case of occupying fewer hard disks, data recovery can be performed in a local scope in the data block recovery process, global data recovery is maximally reduced, reconstruction cost is effectively reduced, and higher fault tolerance is achieved under the same condition.

The embodiment of the invention discloses a coding and decoding method, a code stream, a coder, a decoder and a storage medium, and the method comprises the steps: building a first candidate matching block list of a current block based on template matching; grouping the first candidate matching block list, determining at least two candidate matching block groups, and determining an index value of each candidate matching block group; the coding end determines an index value of a first candidate matching block group corresponding to the current block and writes the index value of the first candidate matching block group into a code stream, and the decoding end determines the first candidate matching block group according to the index value, carries out fusion according to the first candidate matching block group and determines a prediction block of the current block. Therefore, through constructing the candidate matching block list and grouping, indicating the first candidate matching block group according to the index value, and performing fusion prediction by using the first candidate matching block group, the prediction accuracy can be improved, the code rate can be saved, the coding and decoding efficiency can be improved, and the coding and decoding performance is further improved.

The application relates to a comprehensive memory management method and system based on deep learning training, which comprises the following steps: dividing a data set to form a plurality of data blocks; grouping the plurality of data blocks to form a plurality of data block groups, and mapping the data blocks in the same data block group to the same abstract block, so that a plurality of abstract blocks form an abstract memory space in logic; when a data access request is received, determining an abstract block slot corresponding to a data file requested to be accessed; if the abstract block slot is empty or invalid, selecting a target data block from the data block group according to a filling rate of the data block, reloading the target data block, and filling the reloaded content into the empty slot in the abstract block; and marking the actually consumed data file as invalid. The application makes full use of the characteristic that data access has no fixed order in the deep learning training process, and significantly improves the loading efficiency of the training data and the system I / O throughput capacity.

A data storage method, and a storage system, a storage apparatus and a storage device, which are used for improving the utilization rate of a storage space of the storage system. The data storage method comprises: dividing each primary fault domain into a plurality of secondary fault domains, wherein each secondary fault domain comprises some storage apparatuses among a plurality of storage apparatuses, and the storage apparatuses comprised in each secondary fault domain are different from those comprised in one another; dividing data to be written into a plurality of data block groups, wherein each data block group comprises N data blocks, and N is an integer greater than 0; generating at least one primary parity block on the basis of the N data blocks in each data block group, so as to obtain a plurality of groups of primary parity blocks; and respectively storing, in storage apparatuses included in different second fault domains, different data blocks in each data block group, and then generating secondary parity blocks on the basis of the plurality of groups of primary parity blocks, wherein the secondary parity blocks are parity blocks for a plurality of groups of data blocks.

The embodiment of the invention discloses a data storage method, a storage system, a storage device and storage equipment, which are used for improving the storage space utilization rate of the storage system. The data storage method comprises the steps that each first-level fault domain is divided into a plurality of second-level fault domains, each second-level fault domain comprises part of storage devices in a plurality of storage devices, and the part of the storage devices included in each second-level fault domain are different; to-be-written data is divided into a plurality of data block groups, each data block group comprises N data blocks, and N is an integer greater than zero; generating at least one primary verification block according to the N data blocks in each data block group to obtain multiple groups of primary verification blocks; and after different data blocks in each data block group are stored in storage devices contained in different second fault domains, a second-level verification block is generated according to the multiple groups of first-level verification blocks, and the second-level verification block is a verification block of the multiple groups of data blocks.

The present disclosure provides a method, an apparatus, a computer device, and a storage medium for feature extraction of a design draft. Among them, the method includes: obtaining a design draft to be processed corresponding to a page; identifying the design draft to determine at least one block group with a nesting relationship in the design draft and the block feature information of each block in the block group; a block corresponds to a page component in the page; selecting a target block group from at least one block group, and for the target block group, according to the nesting relationship between different blocks in the target block group, fusing the block feature information of the parent block and the block feature information of each child block in the target block group to obtain a target fusion feature corresponding to the parent block; wherein, the target fusion feature is used to classify the parent block to obtain a block classification result of the parent block. The embodiments of the present disclosure can improve the rationality and accuracy of the determined target fusion feature corresponding to the parent block.

This application provides a data compression method, electronic device, and program product. The method includes: obtaining the current load state and predicted load state of a target system; determining a current upper limit value for the number of aggregates based on the current load state and predicted load state; dividing multiple data blocks to be processed into at least one data block group based on the current upper limit value for the number of aggregates, wherein the number of data blocks contained in each data block group is not greater than the current upper limit value for the number of aggregates; performing merging and compression processing on each data block group to obtain compressed data, and persistently storing the compressed data, thereby improving data compression efficiency.

The invention discloses an R3 algorithm-oriented visibility analysis method, system and device and a storage medium, and belongs to the technical field of geographic computing with high performance requirements. The method comprises the following steps: acquiring original elevation data, dividing the original elevation data into a plurality of annular body decomposition domains, and establishing a position mapping relationship between a grid point index and the original elevation data; calculating a feature vector and a memory space of each annular body decomposition domain, and inputting the feature vector into the calculation intensity prediction model to obtain calculation intensity; combining the echoid decomposition domains to obtain a plurality of chunk groups, and combining the echoid decomposition domains in each chunk group to obtain a plurality of block groups; the chunk groups are sent to a GPU end, and an R3 algorithm is executed through the GPU end to obtain a target point result array; and obtaining a final visibility judgment result graph according to the position mapping relationship between the grid point index and the original elevation data. According to the invention, the calculation efficiency of the R3 algorithm is improved.

The invention provides a cluster file system of a database. The cluster file system of the database comprises a shared storage module, a distributed lock management module, a striped metadata layout module, a distributed block allocation strategy module and a lock granularity optimization module. The striped metadata layout module is configured to generate striped metadata in the creation stage of the file and persist the striped metadata to inode metadata of the file, and the striped metadata records the total number of stripes of the file and the block initial address of each stripe; the distributed block allocation strategy module is configured to allocate each strip to a predefined block group or block area in the shared storage module according to the strip number of the file, so as to complete the block allocation of the data to be allocated; and the lock granularity optimization module is configured to construct an independent strip sub-lock for each strip on the basis of the inode file lock of the distributed lock management module, each strip sub-lock independently controls a block area covered by the strip sub-lock, and parallel access of multiple nodes or multiple processes to different strips of the same file is realized.

The application relates to a code rate control method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: obtaining to-be-compressed data and a set target compression rate, and obtaining a total encoding budget after compression; the to-be-compressed data is divided into a plurality of data blocks, and each row of data blocks is grouped to obtain a data block group; an initial value is determined based on the total encoding budget, a current encoding budget of the current data block group and a target encoding budget of a current to-be-compressed data block are obtained; the encoding budgets consumed by each encoding mode of the current to-be-compressed data block are determined, so that the target encoding size of the current to-be-compressed data block is determined; the current encoding budget is updated based on the target encoding size, and the current number of to-be-compressed data blocks remaining in the current data block group is obtained; the target encoding budget of a next data block is obtained based on the current encoding budget and the current number, and the next data block is taken as the current data block until the end. The method can adjust the compression rate.

Embodiments of the present specification provide a data processing method and device, wherein the data processing method comprises: determining a data block group corresponding to initial data, wherein the data block group comprises at least two data block sets; obtaining an initial check block corresponding to the data block group; processing the initial check block corresponding to the data block group based on a target check algorithm to determine a target check block corresponding to the data block group; and replacing the initial check block based on the target check block. In the process of converting the data state of the initial data, only the initial check block corresponding to the initial data needs to be obtained, and the target check block generated based on the initial check block is used to replace the initial check block, thereby reducing the amount of data read and written, avoiding the need to read the entire initial data in the process of data state conversion, reducing the amount of data read and written, and reducing resource consumption.

A data transfer method and apparatus, a host, a storage system, and a storage medium, relating to the technical field of storage. In the method, within a first logical block group of a host, valid data in a first stripe of the first logical block group is transferred to a logical address of invalid data outside the first stripe. The valid data in the first stripe in the first logical block group is transferred, so that the entire first stripe can be freed up. Further, when system garbage collection is performed either manually or automatically by the host, the first stripe is completely emptied for subsequent writing of new data. In addition, only part of the valid data in the first logical block group is transferred without needing to transfer all of the valid data in the entire first logical block group; accordingly, compared to the related art in which all valid data in the first logical block group is transferred to another blank logical block group, the amount of valid data being transferred is reduced, thereby helping to reduce write amplification and performance overhead caused by data transfer.

A video decoding method includes generating an affine merge candidate list including control point-based affine merge candidates corresponding to control point motion vectors when inter prediction of a current block is performed in an affine merge mode, the control point motion vector is determined by using a motion vector of a neighboring block included in a block group of representative neighboring blocks according to a corner of the current block; and determining an affine motion vector of the current block by using a control point motion vector corresponding to a merge candidate selected from an affine merge candidate list, and obtaining a prediction sample point of the current block by using the affine motion vector of the current block, a representative neighboring block adjacent to a lower right corner of the current block is a block for which available motion information is obtained from a neighboring block adjacent to the lower right corner of the current block and a right side boundary of the current block, and a neighboring block diagonally adjacent to the lower right corner of the current block.

The invention provides an earthquake denoising method, system and device based on improved BM3D and a medium, and belongs to the technical field of earthquake denoising, and the method comprises the steps: obtaining a to-be-processed earthquake data profile; calculating a local slope field of the seismic data profile; based on the local slope field and a block matching strategy, grouping similar data blocks in the seismic data profile to form a plurality of three-dimensional block groups; performing event planarization processing on each three-dimensional block group according to slope information extracted from the local slope field; performing de-noising processing on the three-dimensional block group after planarization, and performing inverse planarization processing on the three-dimensional block group after de-noising; and processing each data block subjected to the inverse planarization processing to generate a final de-noised seismic data profile. According to the method, the denoising effect and the signal fidelity of the seismic data of the complex structure are improved.

The invention discloses a pollution factor heterogeneity data-based restoration target area demarcation method and system, and relates to the technical field of restoration target area demarcation, historical pollution data are analyzed, a plurality of historical pollution sub-data are extracted, and each sub-data corresponds to the pollution process of one area; constructing a pollution process expression model to express process characteristics of the pollution factors; classifying the historical models to form a historical pollution process expression model library, performing block splitting on the models in the library to obtain a plurality of blocks, and associating and combining core blocks and adjacent blocks into block groups; comparing the block groups in the same library, and classifying conforming blocks into one class; the method comprises the steps of determining the current pollution condition, calling a corresponding library to generate a current pollution process expression model, randomly locking an edge block, comparing the edge block with a historical block group, and selecting the most conforming block group to explain the subsequent pollution process, and through the technical scheme, the effect of predicting and deducing the soil pollution process is achieved, and the precision of delimiting the remediation target region is improved.