8 results about "Block group" patented technology

The application relates to the computer technical field and discloses a data processing method based on erasure code and related devices. The application obtains a plurality of original data blocks; divides the plurality of original data blocks into two sequential groups and a plurality of cross-group data block groups; the original data block serial numbers in the sequential groups are continuous, each cross-group data block group contains the same number of original data blocks, and the cross-group data block serial numbers are discontinuous; a plurality of check blocks of the plurality of original data blocks are generated, the plurality of check blocks include two global check blocks, sequential group check blocks corresponding to each sequential group, and cross-group check blocks corresponding to each cross-group data block group. Thus, in the case of occupying fewer hard disks, data recovery can be performed in a local scope in the data block recovery process, global data recovery is maximally reduced, reconstruction cost is effectively reduced, and higher fault tolerance is achieved under the same condition.

The application relates to a comprehensive memory management method and system based on deep learning training, which comprises the following steps: dividing a data set to form a plurality of data blocks; grouping the plurality of data blocks to form a plurality of data block groups, and mapping the data blocks in the same data block group to the same abstract block, so that a plurality of abstract blocks form an abstract memory space in logic; when a data access request is received, determining an abstract block slot corresponding to a data file requested to be accessed; if the abstract block slot is empty or invalid, selecting a target data block from the data block group according to a filling rate of the data block, reloading the target data block, and filling the reloaded content into the empty slot in the abstract block; and marking the actually consumed data file as invalid. The application makes full use of the characteristic that data access has no fixed order in the deep learning training process, and significantly improves the loading efficiency of the training data and the system I / O throughput capacity.

This application provides a data compression method, electronic device, and program product. The method includes: obtaining the current load state and predicted load state of a target system; determining a current upper limit value for the number of aggregates based on the current load state and predicted load state; dividing multiple data blocks to be processed into at least one data block group based on the current upper limit value for the number of aggregates, wherein the number of data blocks contained in each data block group is not greater than the current upper limit value for the number of aggregates; performing merging and compression processing on each data block group to obtain compressed data, and persistently storing the compressed data, thereby improving data compression efficiency.

The invention discloses an R3 algorithm-oriented visibility analysis method, system and device and a storage medium, and belongs to the technical field of geographic computing with high performance requirements. The method comprises the following steps: acquiring original elevation data, dividing the original elevation data into a plurality of annular body decomposition domains, and establishing a position mapping relationship between a grid point index and the original elevation data; calculating a feature vector and a memory space of each annular body decomposition domain, and inputting the feature vector into the calculation intensity prediction model to obtain calculation intensity; combining the echoid decomposition domains to obtain a plurality of chunk groups, and combining the echoid decomposition domains in each chunk group to obtain a plurality of block groups; the chunk groups are sent to a GPU end, and an R3 algorithm is executed through the GPU end to obtain a target point result array; and obtaining a final visibility judgment result graph according to the position mapping relationship between the grid point index and the original elevation data. According to the invention, the calculation efficiency of the R3 algorithm is improved.

The invention provides a cluster file system of a database. The cluster file system of the database comprises a shared storage module, a distributed lock management module, a striped metadata layout module, a distributed block allocation strategy module and a lock granularity optimization module. The striped metadata layout module is configured to generate striped metadata in the creation stage of the file and persist the striped metadata to inode metadata of the file, and the striped metadata records the total number of stripes of the file and the block initial address of each stripe; the distributed block allocation strategy module is configured to allocate each strip to a predefined block group or block area in the shared storage module according to the strip number of the file, so as to complete the block allocation of the data to be allocated; and the lock granularity optimization module is configured to construct an independent strip sub-lock for each strip on the basis of the inode file lock of the distributed lock management module, each strip sub-lock independently controls a block area covered by the strip sub-lock, and parallel access of multiple nodes or multiple processes to different strips of the same file is realized.

Embodiments of the present specification provide a data processing method and device, wherein the data processing method comprises: determining a data block group corresponding to initial data, wherein the data block group comprises at least two data block sets; obtaining an initial check block corresponding to the data block group; processing the initial check block corresponding to the data block group based on a target check algorithm to determine a target check block corresponding to the data block group; and replacing the initial check block based on the target check block. In the process of converting the data state of the initial data, only the initial check block corresponding to the initial data needs to be obtained, and the target check block generated based on the initial check block is used to replace the initial check block, thereby reducing the amount of data read and written, avoiding the need to read the entire initial data in the process of data state conversion, reducing the amount of data read and written, and reducing resource consumption.

The invention provides an earthquake denoising method, system and device based on improved BM3D and a medium, and belongs to the technical field of earthquake denoising, and the method comprises the steps: obtaining a to-be-processed earthquake data profile; calculating a local slope field of the seismic data profile; based on the local slope field and a block matching strategy, grouping similar data blocks in the seismic data profile to form a plurality of three-dimensional block groups; performing event planarization processing on each three-dimensional block group according to slope information extracted from the local slope field; performing de-noising processing on the three-dimensional block group after planarization, and performing inverse planarization processing on the three-dimensional block group after de-noising; and processing each data block subjected to the inverse planarization processing to generate a final de-noised seismic data profile. According to the method, the denoising effect and the signal fidelity of the seismic data of the complex structure are improved.