174 results about "Compress Algorithm" patented technology

The invention relates to the technical field of data compression, in particular to a ZSTD compression algorithm optimization method based on joint coding and ternary decomposition, and the method comprises the following steps: S1, building a joint frequency table based on the joint distribution of the literal length and the matching length of a previous section, setting a frequency deviation criterion, and dynamically adjusting a low-frequency combination proportion; s2, decomposing the offset value structure into a leading segment, a trailing segment and a middle segment, performing bit width-entropy ratio analysis on the middle segment, and selecting a symbol index or bit stream coding; and S3, entropy discriminant coding is performed on the joint symbol and the offset symbol respectively, and a structure position is mapped synchronously to generate a structured compressed output stream. According to the method, collaborative optimization of frequency self-adaption, redundancy minimization and structured mapping is realized through joint symbol dynamic modeling and offset ternary structure entropy discriminant coding, so that the ZSTD compression efficiency and the coding stability are remarkably improved.

The invention discloses a memory management method and related equipment, and relates to the technical field of data storage, the method comprises the following steps: based on access information of memory pages of a target operating system, determining a dynamic popularity score of each memory page; on the basis of the current memory pressure level and the dynamic popularity score of the target operating system, executing a corresponding hierarchical data migration strategy so as to migrate different memory pages to the corresponding main Swap storage space or auxiliary Swap storage space; adjusting a compression algorithm applied to the auxiliary Swap storage space based on the current central processing unit utilization rate of the target operating system; and detecting real-time states of the main Swap storage space, the auxiliary Swap storage space and the physical memory, and adjusting a hierarchical data migration strategy and a compression algorithm based on a detection result. Through dynamic popularity perception, hierarchical data migration, adaptive compression regulation and cross-layer collaborative optimization, the memory utilization rate, the exchange efficiency and the overall system stability can be improved under different system loads.

The invention relates to a simulation result compression method and system based on semantic segmentation, equipment and a medium. The method comprises the following steps: performing feature extraction on a simulation result file to obtain semantic features; generating a segmentation point sequence according to the features by using a semantic segmentation model, and segmenting the file into a plurality of semantic blocks; extracting a compressibility feature vector for each block, and dynamically distributing an optimal compression strategy for the block through a prediction model; compressing each block by using the distributed strategy to generate a compressed data block; and finally, constructing a block sensing index based on the block type and the used strategy, and integrating the index and the data block to generate a compressed file. By adopting the method, the defects of limited compression ratio and incapability of supporting rapid random access caused by lack of semantic understanding of a general compression algorithm can be overcome, and by understanding the internal logic structure of the file and applying differential compression, the rapid retrieval and analysis capability of specific data is kept while the high compression ratio is realized.

The invention provides a power plant high-frequency data transmission method based on a lossless compression algorithm. The power plant high-frequency data transmission method comprises the following steps of performing quality preprocessing on original high-frequency data; implementing hierarchical acquisition and periodic packaging according to data real-time requirements; performing differential coding for the data type: performing run-length coding on the switching value, and performing differential coding on the analog quantity; carrying out unified lossless compression on the coded data by using a Zstand-ard algorithm; the compressed data is encrypted and transmitted to an industrial internet platform, challenges confronted by a speed regulation system are handled through the steps, automatic optimization and dynamic adjustment of a data compression algorithm are achieved so as to adapt to the requirements of different application scenes, and the stability and efficiency of power production are improved.

The application discloses an electric energy meter data compression method and system, an electronic device and a storage medium. The method first performs first-order difference processing on an electric quantity measurement data sequence to obtain a residual data sequence. The value range of the residual data sequence is greatly reduced, the repetition degree is greatly increased, and the statistical entropy is significantly reduced, thus creating an ideal input with low entropy and high redundancy for subsequent coding, and improving the compression ratio as a whole. The residual data sequence is then coded, the positive and negative information is retained, and the problem of equivalent entropy not being reduced due to complement storage is avoided. The residual coded sequence is then subjected to bit column reorganization processing or byte column reorganization processing, and a continuous data stream with a large number of locally repeated segments is generated, so that in the subsequent electric energy meter environment with limited MCU computing power and RAM, a lightweight compression algorithm can also quickly find and reuse redundancy in a very short window, and the overall compression efficiency is significantly improved.

A system and method for zero-knowledge verifiable codebook compression receives an input data stream comprising data blocks and encodes the stream using codebook-based compression algorithms. Concurrently with encoding, the system generates zero-knowledge proofs that cryptographically attest that the encoded representation will decode to data having a specified digest and that policy appendices associated with the codebook were applied during encoding. The system generates codebook commitments comprising cryptographic commitments to codebook contents and policy metadata, then formats output packets containing the encoded representation, zero-knowledge proof, and public inputs including the specified digest and codebook commitment. The zero-knowledge proofs enable verification systems to validate encoding correctness and policy compliance without accessing plaintext content or codebook contents.

The invention discloses a multi-modal corpus storage method and device, computer equipment and a medium, and the method comprises the steps: carrying out the compression processing of text modal data and audio and video modal data of a multi-modal corpus, and employing a suitable compression algorithm for the characteristics of different modal data; similarity calculation is carried out on the text compression features, the audio and video compression features and corresponding modal features of the newly-added multi-modal corpus, semantic redundancy of the newly-added multi-modal corpus and historical compression corpus can be recognized, a fusion compression process is triggered to eliminate cross-modal redundancy, and repeated storage of same semantic information is avoided; corpus features which do not trigger the fusion process are independently stored, the corpus features which trigger the fusion process are fused to generate multi-modal joint features, and the stored data volume is further simplified; the corpus features are migrated to the corresponding data storage layer in combination with the access frequency and the response time, so that the storage cost is reduced.

The invention discloses a self-adaptive compression and lossless fast transmission method and system for high-frequency test data. The method comprises the following steps: extracting multi-dimensional dynamic feature vectors from high-frequency test data in real time and marking feature points; based on the features, the network bandwidth and the target compression ratio, dynamically selecting a compression algorithm combination and parameters through a pre-training decision model; performing data framing and typed preprocessing according to the algorithm combination; compression is executed through a multi-stage assembly line, and a correction link based on statistical characteristics is inserted between stages; constructing a three-dimensional transmission state matrix fusing priorities, links and quality scores, dynamically allocating links and adaptively adding forward error correction codes; and after error correction and decompression of a receiving end, realizing lossless verification through multi-dimensional comparison of feature point sequences of the reconstructed data and the original data. According to the method, the whole-course self-adaption of the compression strategy and the transmission scheduling is realized, and the lossless and reliable recovery of the data is ensured while the compression and transmission efficiency is improved.

The invention relates to a configurable decompression circuit supporting COO and Bi tmap compression algorithms, and belongs to the field of integrated circuits. According to the hybrid compression method suitable for the circuit, after sparseness analysis is carried out on weight matrixes of all layers of a neural network, COO compression based on a coordinate type sparse matrix or Bitmap compression based on bitmap masks is selected according to the sparseness characteristics of different layers, so that the storage space is optimized; comprising a control module, a first selector, a second selector, a bitmap description memory, a coordinate index memory, a numerical memory, a converter, a COO decoder and a decompression data storage module, decoding of two compression formats of COO and Bi tmap can be supported, and the operation efficiency and flexibility of a neural network processor are improved. The method effectively reduces the weight storage demand of the neural network, enhances the adaptability of the compression algorithm, and is suitable for hardware implementation of an efficient neural network model.

The invention discloses a hardware acceleration method and system for a ZSTD data compression algorithm based on an FPGA. Unified hardware implementation and collaborative optimization are carried out on three key links of LZ77 character matching, Huffman coding and finite state entropy coding in the ZSTD data compression algorithm. According to the system, a multi-channel parallel LZ77 character matching hardware architecture is adopted, and character matching is executed on continuous positions in a plurality of parallel windows at the same time. A modular design and a deep pipeline structure are adopted, and a buffer and pipeline mechanism is introduced among modules such as an LZ77 matching module, a Huffman coding module and a finite state entropy coding module. The 375MB / s compression throughput can be achieved under the 100MHz clock frequency, the compression speed is remarkably increased while the compression ratio is guaranteed, and the method is suitable for scenes with high requirements for compression performance of data centers, edge computing and embedded systems and has high practicability and popularization value.

The application relates to the field of network security, in particular to an IPSec anti-replay detection method based on multiple caches. The latest window information which has not been stored in DDR is cached by means of the cache space of FPGA. When the sn window information of the previous message is not updated in the algorithm decryption stage, the un-updated sn window information is stored in the cache space of FPGA, the anti-replay detection of the next message is not affected, the processing time of two messages is shortened to the algorithm processing time, the preparation time before the algorithm decryption is greatly compressed, the method has the characteristics of fast speed, less resource occupation and the like, and is suitable for devices which have strict performance requirements and need to perform anti-replay detection.

The present application relates to the field of digital information transmission, in particular to a ship navigation data security evidence method and system. The method comprises: collecting and preprocessing real-time navigation state data of the ship; calculating a navigation maneuverability index reflecting the current maneuvering intensity of the ship by using the preprocessed real-time navigation state data of the ship; determining the length of the sliding window of the compression algorithm based on the navigation maneuverability index through a preset reverse attenuation model, wherein the length of the sliding window is negatively correlated with the navigation maneuverability index; configuring the length of the sliding window as the lookup range of the compression encoder, and performing compression and encryption processing on the real-time navigation state data of the ship to obtain an evidence data packet and write it into a shipborne storage medium. The present application dynamically adjusts the compression strategy, effectively balances the storage space utilization rate and the consumption of computing resources while ensuring real-time data evidence, and improves the transmission and storage efficiency of ship navigation data.

The application relates to the technical field of computing power allocation, and provides a computing power allocation method and system supported by an edge collaboration of a cloud computing platform, which comprises an architecture of deploying a complete dialogue model by a center node, carrying a light node model by an edge node, and configuring a semantic recognition module by a local node, the local node generates a conversation constraint set through a semantic recognition and semantic compression algorithm, long conversation model omission of user early demands is avoided, and dialogue accuracy is improved. In addition, a semantic complexity coefficient is obtained by analyzing the constraint set, the edge node collects self-computing power related parameters to evaluate node states, divides computing power rich and tight nodes, the computing power tight node processes low complexity tasks according to the self-computing power margin, the rest of the tasks are scheduled to the center node or adjacent computing power rich nodes, the tasks are distributed according to the complexity coefficient, the edge node computing power is fully utilized, and the overall computing power utilization efficiency is improved.

The invention discloses a lossless compression method and device for estimating a compression ratio of a compression algorithm based on a time sequence data characteristic value in a database. The method comprises the following steps: acquiring time sequence data blocks grouped according to columns; extracting characteristic values of the data blocks; aiming at a plurality of lossless compression algorithms, calculating each algorithm estimation compression ratio based on a preset compression ratio estimation formula and the characteristic value; selecting the algorithm with the highest estimation compression ratio as a target algorithm; and performing lossless compression on the data block by utilizing the target algorithm. According to the invention, through a lightweight estimation mechanism, high cost and high overhead of an AI model are avoided, and adaptive compression is realized with low calculation overhead; the time sequence data compression ratio is effectively improved (the time sequence data is close to the theoretical limit when the potential data is high and reaches the highest compression ratio of 75% when the potential is low), the compression potential is fully explored, the storage cost is remarkably reduced, and high efficiency and practicability are achieved.

The application provides a compression parameter analysis calculation method and system of an LZ77 compression algorithm, comprising judging whether an LZ77 code of compressed data stream adopts a flag bit method or an escape code method; for the flag bit method, using obtained flag bit B, bit number D of distance code and bit number L of length code to decompress data stream S to obtain compressed data stream Q, and determining that the bit number of distance code is L; for the escape code method, analyzing distance code with distance K and extension code; in response to K>L, generating special compressible data with distance 1-4 to inject an LZ77 compressor to obtain compressed data, and calculating distance escape code and extension code with distance 1-4. The method calculates LZ77 compression parameters according to enumeration, so as to realize data decompression, solve the problem of decompression of a black box compressor, and has great innovation and practical significance in the application of data analysis.

The application provides an intelligent training acceleration method based on data dense characteristics in a neural network, which collects relevant data information and performance model information by pre-training a model, and dynamically reduces the data amount required to be transmitted between a GPU and a CPU by using the dense characteristics of a deep learning network in combination with compression and decompression algorithms, so as to improve the overall performance of the system. The application also provides an intelligent training acceleration system based on data dense characteristics in a neural network, which comprises a data acquisition module, a dense data compression rate decision module, a selective compression decision module and a training module; and the neural network training is carried out by using the system, so that the training speed and the overall performance of the system can be effectively improved.

The present application relates to the field of computer graphics animation, and more particular to a system and method for geometric cache optimization. One aspect of the present invention relates to a method for optimizing a geometric data cache. The method comprises importing geometric data into a Universal Scene Description framework, applying an advanced compression algorithm to the geometric data cache, indexing the compressed geometric data, selecting a portion of the compressed geometric data for decompression, and decompressing the compressed geometric data.

The invention discloses a multi-feature adaptive data compression method and system for a power distribution network terminal, and the method comprises the steps: extracting a multi-dimensional feature vector of to-be-compressed data, inputting the multi-dimensional feature vector into a classification module, and obtaining the type of the to-be-compressed data; determining a candidate compression algorithm type according to the type of the to-be-compressed data; calculating the score of each compression algorithm instance in the candidate compression algorithm type to the to-be-compressed data; and selecting the compression algorithm instance with the highest score value to compress the to-be-compressed data. According to the method, fine classification is carried out on the power distribution network data in consideration of multiple dimensions, the optimal compression algorithm is selected in combination with data features, and the service requirements, the compression effect and the compression efficiency are effectively considered.

The invention specifically relates to a logistics digital warehouse management method, and relates to the technical field of warehouse logistics management, and the method comprises the steps: abstracting a warehouse into a weighted directed graph, constructing an AGV kinematics parameter table, converting a physical inertial parameter into a node access cost, and completing the mapping and initialization of a model. According to the method, warehouse physical information and AGV inertial parameters are converted into digital models which can be calculated by a computer through a weighted directed graph and an AGV kinematics parameter table, and mapping faults of physical and digital layers are eliminated; time is used as an independent dimension to be fused into a topological structure, and multi-AGV space-time conflicts are avoided from the source; meanwhile, reachability and kinematics memory level pruning are adopted, invalid nodes are directly removed, invalid edges are cut off, the algorithm search space is greatly compressed, and the problem of high calculation complexity caused by node explosion is solved.

The invention relates to the technical field of information, in particular to an artificial intelligence decision support method and system for high-frequency transactions. The method comprises the steps of collecting order book data, and obtaining a fluctuation rate and a trading price difference based on the order book data; the window at the current moment is adjusted based on the volatility to obtain a window at the next moment, and window detection is completed through the hash value; counting the order quantity difference of the key prices at the current moment and the adjacent moment and the difference between the buying and selling price difference at the current moment and the maximum buying and selling price difference for the window with the data change, and obtaining a dynamic compression ratio based on the two differences; the DPCM algorithm and the dictionary compression algorithm are adjusted based on the dynamic compression rate, and then compression is carried out; and generating a decision signal based on the compressed data to complete decision support. The compression speed is increased, and the accuracy and integrity of the data are improved.

The invention discloses a hardware-friendly Transform column balance pruning model compression and efficient deployment method. A model compression algorithm, a lightweight parameter storage format, an operation data buffer, a systolic array operation block, a vector operation unit, a nonlinear operator unit, a data flow controller and a DMA unit are included. A model compression algorithm and an efficient deployment architecture are explored according to Transform network software and hardware collaborative reasoning requirements: in a software level, the scale calculation complexity of model parameter quantities is reduced through a fine-grained column balance structured pruning strategy, and parameters are stored in a single-instruction multi-data-stream format and the parameter storage efficiency is optimized through mask code storage; according to the hardware level, an edge computing-oriented Transform special accelerator architecture is designed, so that the architecture can support column balance structured pruning characteristics and a lightweight parameter storage scheme in an original manner. According to the Transform model compression and efficient deployment method, the parameter sparsity after structured pruning is fully utilized, so that the parameter storage pressure of a hardware architecture is reduced, the complex balance of an arithmetic unit is ensured, the operation efficiency of an accelerator is improved, and load balance and efficient reasoning during software and hardware collaborative optimization are realized; the method is widely applicable to efficient deployment scenes of Transform models for edge calculation.

The present disclosure provides a compiling method and device, first, obtaining a to-be-compiled code, and generating a plurality of compiling tasks corresponding to the to-be-compiled code; then, for any compiling task, determining a target code segment corresponding to the compiling task before starting to execute the compiling task; then, searching for a cached compressed compiling product corresponding to the target code segment, if the compressed compiling product is successfully found, determining a target decompression algorithm matching the compressed compiling product from a plurality of to-be-selected decompression algorithms; secondly, decompressing the compressed compiling product by using the target decompression algorithm to obtain a decompressed compiling product corresponding to the compiling task; finally, generating an executable file corresponding to the to-be-compiled code based on the obtained decompressed compiling product. The present disclosure embodiment realizes diversified selection of compression algorithms, improves the flexibility of the selection of compression algorithms during compiling, and improves the compiling efficiency.

This application discloses a data processing method, apparatus, solid-state drive, and storage medium, applicable to fields such as data storage and data retrieval in computer technology. The method includes: determining data to be migrated from compressed data stored on an SSD; determining a first compression algorithm corresponding to the data to be migrated; decompressing the data to be migrated based on the first compression algorithm to obtain first decompressed data; compressing the first decompressed data based on a second compression algorithm to obtain target compressed data; and storing the target compressed data on the SSD; wherein the compression ratio of the second compression algorithm is lower than that of the first compression algorithm. Using this application embodiment can save SSD storage space and improve applicability.

A bus control unit includes: a compression execution unit, configured to compress sparse data by using a compression algorithm, to obtain compression data and compression attribute information; and a compression control unit, configured to generate, according to an instruction format of a communication bus, a data signal carrying the compression attribute information and the compression data, and send the data signal through the communication bus, where the compression attribute information is used for decompressing the compression data to obtain the sparse data.

The application discloses a kind of water conservancy video monitoring safety protection method, system and equipment, and wherein method includes: the smart contract management equipment authentication of alliance block chain architecture, the distributed security access control of decentralization;According to the space-time feature and information entropy analysis of video content Selection adaptive compression algorithm, real-time adjustment compression parameter;Based on the dynamic key derivation algorithm of block chain consensus mechanism and high-precision timestamp, construct double-layer encrypted video transmission mechanism;Based on the distributed hierarchical storage strategy of block chain, execute the hybrid mechanism of local cache and block chain distributed storage;Build timeliness cloud edge collaborative control mechanism.The application realizes the safe access, efficient processing, encrypted transmission, anomaly detection and reliable storage of video monitoring data through the deep integration of edge computing and block chain technology, significantly improves the data credibility, tamper resistance and overall system security of reservoir safety monitoring.

The invention discloses a memory compression method and device, and the method comprises the steps: determining a target compression algorithm corresponding to a to-be-compressed memory page in a plurality of compression algorithms; recording a corresponding relation between the to-be-compressed memory page and the target compression algorithm in a memory compression module; and when the memory compression module is in the first mode, obtaining a target compression algorithm according to the corresponding relationship, and performing compression processing on the to-be-compressed memory page based on the target compression algorithm.

The application relates to the technical field of data processing, in particular to a concrete test piece quality inspection data compression management method, which comprises the following steps: numbering the quality inspection data of each concrete test piece according to production time, and determining the neighborhood data of each concrete test piece; determining the data sequence of each concrete test piece according to the quality inspection data and the neighborhood data of each concrete test piece, determining the first average value of the data sequence, modifying the first average value based on the first type data greater than the first average value and the second type data smaller than the first average value in the data sequence, and obtaining a modified average value; determining the outlying degree of the quality inspection data of each concrete test piece, and determining the quality inspection data with an outlying degree greater than a threshold value as outlying data; and compressing other data except the outlying data in the quality inspection data of each concrete test piece based on a sliding window trajectory compression algorithm to obtain compressed data. Thus, the compression effect of the quality inspection data of the concrete test piece is improved.

An apparatus is arranged to generate a data signal comprising compressed data for a biomedical waveform signal and an indication of a property of the compression process used to generate the compressed data from the biomedical waveform signal. The indication of the property of the compression process is indicative of a compression algorithm used to generate the compressed data A second apparatus includes a receiver (201) receiving the data signal. A decompressor (303) generates a reconstructed biomedical waveform signal by decompression of the compressed data. Further, the second apparatus comprises an adapter (307) which adapts the decompression in dependence on the indication of the property of the compression by selecting a decompression algorithm from a set of decompression algorithms including at least a wavelet transform decompression algorithm and a non-wavelet decompression algorithm. The approach may provide a highly efficient and flexible compression of biomedical waveform signals.

The invention discloses a KV cache intelligent scheduling and semantic structure compression method and system based on a knowledge graph, and the method comprises the steps: firstly, extracting entities and relationships from an input text to construct the knowledge graph, and then employing a graph embedding and structure compression algorithm to aggregate nodes with similar semantics into a high-density semantic KV item, so as to eliminate redundancy, and improve the KV cache intelligent scheduling and semantic structure compression efficiency. And then constructing a semantic scheduling graph, and dynamically loading the most relevant semantic sub-graph cache through a semantic migration mechanism. According to the method, the problems of semantic redundancy, lack of structural understanding, poor dynamic adaptability and the like in a traditional KV cache are effectively solved, the cache utilization rate, semantic consistency and long-range context processing capacity are remarkably improved, and the method is suitable for scenes such as large language model reasoning acceleration and intelligent dialogue.

The application discloses a performance equalization compression method of satellite images. The method equalizes the excellent compression performance of images and the poor compression performance of images from the perspective of a multi-image system by arranging the images, overcomes the influence of the barrel effect on the practicability of image compression, overcomes the defect that the compression quality of a plurality of images compressed by a JPEG2000 standard compression algorithm is too different, and improves the performance of the standard compression algorithm without increasing complexity. The method has the characteristics of easy software and hardware upgrading and realization, has practical value in a satellite image compression transmission system, and can also play a role in any other multi-source standard compression transmission occasion.