1114 results about "Compression method" patented technology

The invention provides a context compression method based on multi-round dialogue intention graph construction, which comprises the following steps: S1, dialogue data acquisition and preprocessing: carrying out natural language processing on each dialogue unit; s2, intention atlas construction is achieved through node design and edge design, and each node comprises original text content and structured semantic information; s3, carrying out context compression and graph structure cutting, and only retaining sub-graphs forming a core semantic link; s4, dynamic context management and topic jump processing: in a multi-topic dialogue, when a user jumps or switches to a new topic, a system records a sub-graph of a current active topic, and contextual nodes of an inactive topic are frozen; s5, context sequence generation and model input: linearizing node contents in the cut sub-graph according to a dependent link sequence to generate a compressed context sequence, and transmitting the context sequence and current user input to a large language model for reasoning; and S6, continuous updating and feedback optimization are carried out.

The invention relates to the technical field of data compression, in particular to an intelligent data compression method and system based on multi-protocol heterogeneous equipment interconnection. The method comprises the following steps: when it is detected that a new heterogeneous device applies to join a network, carrying out communication protocol real-time analysis and protocol conversion adaptation to obtain a heterogeneous device adaptation protocol; carrying out multi-device data stream receiving according to a heterogeneous device adaptation protocol, and generating data streams of a plurality of time windows; adaptive coding and multi-level compression are carried out on the data streams of the multiple time windows, and a multi-device adaptive compression mechanism is constructed; identifying compression load characteristics of each heterogeneous device, and performing dynamic task allocation to obtain an intelligent task allocation strategy; and driving real-time data intelligent compression operation based on an intelligent task allocation strategy and a multi-device adaptive compression mechanism. According to the invention, efficient and adaptive data compression of different devices is realized, and the data transmission efficiency and quality are improved.

The invention relates to the technical field of model compression, and provides a large model compression method and device, a task processing method and equipment and a storage medium, and the large model compression method comprises the steps: carrying out the layer-by-layer quantification of a linear layer of a to-be-compressed initial large model, and obtaining a first large model; the initial large model is a pre-trained large language model constructed based on an expert hybrid architecture; performing route calibration on each expert sub-model in the first large model to obtain a second large model; in the reasoning process of the second large model, based on the task type of a to-be-executed target task, the importance of each expert sub-model in the task type is evaluated; and performing dynamic pruning on each expert sub-model based on importance so as to compress the second big model. Through a compression mode of combining static quantification and dynamic pruning, on the basis of ensuring the model performance, the memory and calculation overhead required by large model reasoning can be reduced, and efficient operation of the large model on light-weight equipment with limited video memory resources is facilitated.

The invention provides a real-time video image compression method based on deep learning, and relates to the technical field of video image compression, and the method comprises the steps: carrying out the key feature recognition through employing an attention mechanism; performing convolution training optimization on the video image sample data set by using a deep learning network structure; a self-encoder structure is designed to carry out feature map encoding compression; a video image compression adaptive network is generated through series fusion; a real-time video image frame is collected for preprocessing, and feature compression processing is performed on a standard video image frame based on a video image compression adaptive network. According to the method and the device, the technical problem that the video compression quality is reduced due to the fact that the generalization ability is insufficient in the face of various scenes and the video compression strategy is difficult to adaptively adjust according to different scenes in the prior art can be solved, the adaptive network is constructed through the combination of deep learning and the auto-encoder, and the video compression quality is improved. And the video compression strategy is dynamically adjusted according to the contents of different video images, so that the video compression quality is improved.

The invention provides a data compression method and device, and relates to the technical field of data processing and data compression. The method comprises the following steps: acquiring to-be-compressed data, and acquiring a data category of the to-be-compressed data; determining a plurality of candidate compression strategies corresponding to the to-be-compressed data based on the data category; the importance degree of the to-be-compressed data is obtained, and the equipment state parameter of the current equipment is obtained; determining a target compression strategy and a compression level corresponding to the target compression strategy from the candidate compression strategies based on the equipment state parameters and the importance; and executing the target compression strategy on the to-be-compressed data according to the compression level to obtain compressed data. According to the method, the target compression strategy can be dynamically determined according to the data type, the data importance and the equipment state, high flexibility and adaptability are achieved, the data storage and processing efficiency is improved, good balance between the data quality and the storage space is ensured, and the method adapts to modern changeable equipment environments and data requirements.

Methods and apparatus are provided for encoding and subsequent decoding of video data by using a learnt video specific dictionary for implicit neural compression. An implicit neural representation comprising a head layer and a tail layer is used with approximations to the head layer parameters. The approximations are determined with combinations of atoms of a learnt video specific dictionary. In one embodiment, the head layer approximations and tail layer parameters are encoded in a bitstream. The dictionary is learnt at the decoder. In another embodiment, a dictionary is sent in the bitstream. At decoding, a reconstructed image is computed using transmitted INR parameters.

Methods and systems implement a pleno-generation face video compression framework with bandwidth intelligence for generative models and compression. Heterogeneous-granularity facial description regularizes long-term dependencies between video frames and compensates for motion estimation errors caused by compact representations of motion information. A generative decoder reconstructs heterogeneous-granularity visual representations, providing auxiliary visual signals for attention-based recalibration of a GFVC-reconstructed face signal. A coarse-to-fine generation strategy avoids error accumulation. High efficiency for heterogeneous-granularity signal compression is achieved by two different entropy-based signal compression methods: heterogeneous-granularities feature representation from the key-reference frame as hyperpriors to optimize the entropy model for compressing heterogeneous-granularity feature from subsequent inter frames, and a feature difference operation for heterogeneous-granularities feature representation between key-reference and subsequent inter frames, such that the entropy model only compresses heterogeneous-granularities feature residual for redundancy reduction. Mixed-model dataset generation and training and model-specific dataset generation and training are also provided.

The invention provides an unstructured data compression method and device, equipment and a storage medium, and the method comprises the steps: carrying out the feature extraction and clustering processing of a to-be-compressed unstructured data set, and obtaining a data segment clustering group; performing semantic region segmentation processing on each data fragment in the data fragment clustering group to obtain a semantic important region and a semantic non-important region; performing differential coding processing on the semantic non-important regions in the same clustering group to obtain differential coding data; and performing high-quality compression processing on the semantic important regions in the same clustering group, and performing merging processing on the compressed important semantic data and the difference coding data to obtain an unstructured data compression result. According to the method, semantic region segmentation and differential compression processing are performed on the unstructured data, so that the data compression efficiency can be effectively improved on the premise of ensuring the quality of important information, and an efficient solution is provided for storage and transmission of large-scale unstructured data.

The invention discloses a spatio-temporal data compression method and system based on lightweight processing, and relates to the technical field of image data processing, and the method comprises the steps: based on spatio-temporal data to be compressed, establishing standardized data entries, forming a window set through time axis segmentation, carrying out the periodic discrimination of windows according to the frequency domain energy distribution, and constructing spatio-temporal blocks, establishing an error control parameter table based on the global error budget; generating a time coding stream, performing integerization on the three-dimensional coordinates and the attribute values, generating a space coding stream through double difference and run length coding, performing edge folding lightweight and texture compression on the grid data, and generating a joint coding result; and based on a joint coding result, dividing the compressed data into a base layer and a plurality of enhancement layers, and establishing a relevance hierarchical storage structure and a block-level index table to generate the compressed data capable of being transmitted in a streaming manner. According to the method, collaborative compression can be carried out by utilizing spatial-temporal data internal relevance.

The invention relates to the technical field of artificial intelligence and big language model reasoning optimization, and discloses a big language model-oriented adaptive KV cache compression method and system, and the method comprises the steps: constructing a lexical element importance measurement mechanism; analyzing an attention head distribution structure in large language model reasoning, and constructing a plurality of pruning strategies; based on a lexical element importance measurement mechanism and the attention head distribution structure, designing a self-adaptive key value cache compression hybrid strategy set based on a pruning strategy; constructing a static self-adaptive key value cache compression method, and automatically distributing a key value cache compression strategy in a pre-filling stage of large language model reasoning; and in a decoding stage, performing adaptive compression on the key value cache based on the allocated key value cache compression strategy. According to the method, the key value cache can be efficiently compressed on the premise of not depending on explicit attention score calculation, a system-level reasoning optimization framework is compatible, the generation performance is kept, meanwhile, the video memory consumption is remarkably reduced, and the long context reasoning capability is enhanced.

The invention provides a neighbor projection type gradient coordination compression method and device and a federated learning system, and relates to the technical field of federated learning. According to the method, the uploading gradient of the client side is obtained, direction normalization is carried out, direction embedding updating of the client side is carried out through index moving average, then the similarity between the client sides is calculated, a neighbor set is generated, a similar graph is constructed, and then conflict detection and weight setting are executed. And performing weighted orthogonal projection in the frozen neighbor direction of each client to correct the gradient, and finally performing weighted aggregation on the gradient and updating global model parameters. According to the method, gradient conflicts are efficiently detected and corrected in a local range by constructing the client similar graph and combining a neighbor projection mechanism, the problems of high complexity and excessive information reduction caused by global processing are avoided, the convergence speed, stability and precision of a global model are improved under the condition that additional calculation and communication overhead of the client is not increased, and the user experience is improved. The method is suitable for large-scale non-independent identically distributed data scenes.

A system and method for optimizing data compression and encryption using reinforcement learning. The system analyzes incoming data streams to extract statistical features and data characteristics, which are processed by a reinforcement learning engine to automatically configure a multi-stage compression pipeline. Each compression stage transforms data into optimized distributions, applies Huffman coding, and maintains full encryption using homomorphic operations. A performance monitor tracks compression efficiency, processing speed, and output quality in real-time, providing feedback to continuously improve the reinforcement learning model's decisions. The system can dynamically adjust between one to five compression stages and select appropriate compression methods, including traditional algorithms or neural network-based approaches, based on data characteristics and performance requirements. All processing occurs on encrypted data without requiring decryption, ensuring complete data security throughout the pipeline. The adaptive nature of the system enables optimal compression performance across diverse data types while maintaining encryption integrity.

The invention relates to the technical field of big language model dialogue system optimization, in particular to a big language model dynamic dialogue history compression method and system.The method comprises the steps that the maximum length of a context window matched with a target big language model, the maximum number of newly-generated lexical elements and the size of a safety buffer area are set, and then dialogue history is loaded; initial compression and verification are carried out through a keyword and TF-IDF mixed scoring system, multiple times of dynamic compression are carried out according to gradients if the conditions are not met, and finally, parameters are adjusted to adapt to the residual space when a model is called to generate response. The system comprises a dialogue history loading module, a parameter configuration module, a dynamic compression engine module and a large language model integration module. Through a multi-stage compression verification mechanism and a progressive multi-stage dynamic compression strategy, super-long texts such as engineering technology documents can be processed, service interruption is reduced, the compression efficiency is improved on the premise that key semantics are reserved, and multi-language dynamic compression is supported. The problems of system token overrun and service instability in the prior art are solved.

The invention provides a key value cache fusion compression method and device, electronic equipment and a storage medium, and the method comprises the steps: selecting an adjacent historical query as an observation window based on the position of a current query, and observing all historical cache key value pairs corresponding to a plurality of historical queries; determining target attention weights of all historical cache key value pairs in the observation window; based on each target attention weight, selecting a key value pair of a reserved cache; and performing compensation reconstruction on the value vector in the to-be-expelled key value pair, and fusing the reconstruction compensation component of the to-be-expelled key value pair and the key value pair with the reserved cache to obtain a fused and compressed key value pair. According to the key value cache fusion compression method provided by the invention, effective screening and fusion compression of historical cache key value pairs are realized, the problem of context information loss is effectively avoided, the calculation efficiency and the expression ability are considered, the overhead of storage and calculation resources is remarkably reduced while the context integrity is ensured, and the method is suitable for popularization and application. The method is especially suitable for long text processing tasks.

In one implementation, we propose a scalable framework where a base layer uses NN-based methods to compress the content for computer vision machine tasks and enhancement layer(s) use traditional predictive coding for human viewing. Typically, the based layer performs NN-based analysis to generate a latent tensor, which is entropy coded to produce the base layer bitstream. By performing synthesis on the latent tensor, an inter-layer predictor can be obtained for the enhancement layer(s). Since many machine tasks are not required to be performed for each frame, the base layer may skip analysis for some frames. The synthesis may be performed at the base layer or the enhancement layer(s). In one example, the base layer compresses features optimized for a machine task and the enhancement layer(s) rely on predictive coding. In another example, the enhancement layer(s) can use traditional scalable video compression methods.

The invention provides a visual language model binarization compression method and system for an image-text understanding task. The method comprises the steps that the structure of a visual language model for the image-text understanding task is divided into an image coding module, a text coding module and a cross-modal fusion module; dividing the network hierarchy in each module into a plurality of compressible structural units; and carrying out binarization processing on the structural unit in each module, and replacing the original structural unit with the structural unit subjected to binarization processing to obtain a visual language model formed by the compressed image coding module, the text coding module and the cross-modal fusion module. The method further comprises a cross-modal semantic preserving mechanism design. According to the method, the semantic expression capability and task performance of the model are maintained to the maximum extent while the storage and calculation overhead of the model is remarkably reduced, so that the efficient and deployable image-text understanding inference system of the visual language model in a resource-constrained environment is realized.

The invention provides a Token compression method and device for large model dialogues, which can effectively reduce the number of Tokens (lexical elements) of texts input into a large model in multiple rounds of dialogues, and can also retain key information of early parts of historical dialogue texts at the same time. The method is applied to a Token compression management agent, and comprises the following steps: after obtaining a question text input by a user to a terminal, determining the number of lexical elements in an initial text (including a historical dialogue text and the question text) to be sent to a large model; if the initial text lexical element number does not meet the threshold requirement, switching the state into a lexical element compression state, interacting with the large model to obtain the lexical element number of compressed intermediate dialogue texts (parts except for the previous k rounds of dialogue texts, k is greater than or equal to 0), and keeping a semantically generated abstract text. And splicing the first k rounds of dialogue texts, abstract texts and question texts to obtain a target text. If the number of the target text lexical elements meets the threshold requirement, sending to the first large model.

The invention relates to a simulation result compression method and system based on semantic segmentation, equipment and a medium. The method comprises the following steps: performing feature extraction on a simulation result file to obtain semantic features; generating a segmentation point sequence according to the features by using a semantic segmentation model, and segmenting the file into a plurality of semantic blocks; extracting a compressibility feature vector for each block, and dynamically distributing an optimal compression strategy for the block through a prediction model; compressing each block by using the distributed strategy to generate a compressed data block; and finally, constructing a block sensing index based on the block type and the used strategy, and integrating the index and the data block to generate a compressed file. By adopting the method, the defects of limited compression ratio and incapability of supporting rapid random access caused by lack of semantic understanding of a general compression algorithm can be overcome, and by understanding the internal logic structure of the file and applying differential compression, the rapid retrieval and analysis capability of specific data is kept while the high compression ratio is realized.

The invention discloses a signal feature compression method based on codebook discrete quantization and multi-task learning, and belongs to the technical field of crossing of signal processing and artificial intelligence. In order to solve the problem that in the prior art, compression efficiency, semantic retention and calculation complexity are difficult to consider in high-dimensional IQ signal compression, high-dimensional continuous features of an original IQ signal are extracted through an encoder; carrying out vector quantization by utilizing the learnable codebook to generate a discrete index and calculating quantization loss; inputting the discrete features into a decoder branch reconstruction signal to calculate reconstruction loss, and inputting a classifier branch prediction category to calculate classification loss; processing traditional features and coding features by adopting a multi-layer perceptron, and calculating comparison loss through cosine similarity; combining optimization quantization loss, reconstruction loss, classification loss and comparison loss to train a model; and finally outputting a discrete index as a compression feature. The method realizes efficient compression and classification semantic reservation, and is suitable for wireless communication, Internet of Things and other scenes.

The invention discloses a voice data compression and index storage method based on a voiceprint template, and relates to the technical field of voice compression processing. The voice data compression and index storage method based on the voiceprint template comprises the following steps: S1, collecting and preprocessing acoustic state data, coded data and edge synchronization data, and constructing a standardized voice input data set; s2, evaluating the individual sound channel difference degree, and stripping non-universal components highly related to individuals in the voice; s3, analyzing the audio compression content, and constructing a structured total compression amount; s4, performing joint evaluation on real-time transmission regulation and control requirements of the compressed data, and dynamically adjusting an uploading priority and a continuous transmission strategy; and S5, constructing a multi-dimensional reverse index system at the cloud. The problems that for a high-noise far-field pickup scene, an existing compression method does not consider that individual sound channel characteristics are stripped from original signals, so that compression redundancy is high, and retrieval interference is large are solved.

The invention relates to the field of traffic storage optimization and traffic data compression, in particular to a multithreading-combined lossless compression method for fine-grained traffic, which comprises the following steps of: acquiring a traffic data file, extracting a header triple based on the traffic data file, generating an identifier according to the header triple, and transmitting the identifier to a server; aggregating the traffic data files with the same identifier to obtain data streams, and generating a sorted data stream size table according to the data volume of the data streams; creating threads according to system computing power, distributing the data flow to the thread with the minimum thread load according to the data flow size table and the thread load condition, and outputting a data flow distribution scheme; and performing fine-grained characterization and serialization on each data stream to obtain an integer sequence and a byte sequence, selecting a compression processing method according to redundancy characteristics of the integer sequence and the byte sequence, obtaining a redundancy-eliminated sequence, and writing the redundancy-eliminated sequence into a compressed file. The invention aims to provide high compression efficiency to reduce the storage cost while ensuring the data precision.

The invention discloses a compression method and device for generic semiconductor industry operation and maintenance data, electronic equipment and a medium, and relates to the technical field of data compression. The method comprises the following steps: acquiring to-be-compressed generic semiconductor industrial operation and maintenance data; identifying the time sequence of the generic semiconductor industrial operation and maintenance data to be compressed, and judging whether the generic semiconductor industrial operation and maintenance data is time sequence data or not according to the time sequence; and if the to-be-compressed generic semiconductor industrial operation and maintenance data is time series data, splitting the to-be-compressed generic semiconductor industrial operation and maintenance data into a time data set and a feature data set, and compressing the time data set and the feature data set respectively. According to the compression method, different compression strategies are adopted for different types of data, the compression efficiency of the data is improved, and the balance between the compression rate and the calculation efficiency is optimized.

The invention discloses a multi-source heterogeneous time series data compression and motor fault prediction method and system, and the compression method achieves the intelligent compression and feature enhancement of time series data through the cooperation of multi-scale feature extraction, SE weighted fusion, mixed pooling and aggregator collapse. According to the architecture, reasonable reduction of the sequence length and effective improvement of the feature depth can be completed at the same time, on the premise that key fault features are reserved, the technical problem of multi-source heterogeneous sensor data fusion in an industrial scene is solved, and the limitation of serious information loss of a traditional dimension reduction method is overcome.

The invention relates to the technical field of machine learning, and discloses a feature disassembly and compression method and device, equipment and a storage medium, and the method comprises the steps: obtaining an initial feature of to-be-predicted image data, determining an expert activation weight corresponding to a downstream prediction task based on a hybrid expert network, the initial features are processed through a plurality of expert networks in the hybrid expert network to generate disassembled features, the plurality of expert networks are networks constrained through a low-rank matrix, the disassembled features are quantized to generate quantized features, entropy coding is carried out on the quantized features through a hyper-prior network, and compressed features are obtained. According to the method, the expert activation weight is dynamically adjusted along with task requirements, different task scenes are adapted, initial features are disassembled based on a plurality of expert networks, task irrelevant information is explicitly stripped, task key features are reserved, redundant data volume is reduced, quantized coding is performed on disassembled feature input, code stream self-adaption is realized, and bandwidth occupation is reduced.

The invention relates to a KV cache compression method based on attention alignment, and belongs to the technical field of large language models. Comprising the following steps: a target model self-generates dialogue data reply, and constructs training data; adding a soft token into a word list of the target model, and carrying out soft token random initialization; finely adjusting specified parameters, splicing soft tokens at the tail of an original input sequence, and transmitting the soft tokens to the model; and respectively calculating the soft token and the attention distribution of the self-generated reply, calculating the mean square error of the soft token and the attention distribution of the self-generated reply as a loss function, and completing training. According to the method, a limited number of soft tokens playing an auxiliary role are introduced into original input, and importance discrimination of KV cache elements is realized and expelling is completed by aligning attention distribution of the soft tokens and real generated content; according to the method, better compromise is achieved between sequence length compression and performance loss, and the loss of model performance is better controlled while it is guaranteed that KV cache video memory space occupation is reduced.

The invention discloses a multi-modal large model adaptive video frame compression method and system, and relates to the field of multi-modal video analysis, and the method comprises the steps: S1, obtaining a user text instruction and a sampling video frame of an original video; s2, converting the user text instruction into a space-time semantic instruction through hierarchical thinking chain reasoning; s3, extracting visual features of the sampled video frames, and performing importance scoring on the visual features through a space-time semantic instruction to obtain a semantic weight matrix; and S4, based on the semantic weight matrix, dynamically adjusting the number of visual features and the spatial resolution of each frame, and based on the new spatial resolution, adjusting adaptive pooling parameters and performing adaptive weighted pooling to obtain compressed and refined features. According to the method, a user text instruction is decoupled into a time, space and context three-dimensional instruction, a dynamic semantic weight matrix is generated, and a vision-text semantic alignment error is reduced; the token density is adaptively adjusted based on the weight matrix, the redundant region is compressed and merged, and the calculation complexity is reduced.

The invention belongs to the field of text questioning and answering, and relates to a large language model long text questioning and answering method and system based on a hybrid context compression technology. The method comprises the following steps: analyzing and preprocessing a document, and converting an unstructured original text into a normalized text paragraph set; classifying questions of the long text question and answer scene based on a large language model; according to the question type and the preprocessed text, adaptively selecting the most suitable context compression method to compress the long text to obtain a compressed context; and generating an answer to the question based on a large language model by using the compressed context. According to the method, the advantages of two context compression technologies are integrated, the problem of context window limitation of processing a long text by a large language model is effectively solved, high-quality question and answer performance is kept while computing resource consumption and processing delay of the compression technologies are reduced, and the limitation of a single compression method on different types of problems is relieved.

The invention discloses an entropy coding compression method for high-dimensional sparse data, which relates to the technical field of data compression, and comprises the following steps: reading an original high-dimensional sparse matrix, extracting a position index set and a corresponding non-zero value set of all non-zero elements, extracting active samples from the non-zero value set, and compressing the active samples. After an active sample matrix and an optimal mean value centralization matrix are constructed, principal component projection and a self-expression structure are introduced for joint modeling, a low-rank robust optimization objective function is formed, and a principal component feature matrix is finally output by alternately optimizing mean values, projection, weights and residual errors. According to the method, the compression efficiency and the processing pertinence of high-dimensional sparse data are effectively improved, self-expression structure modeling and residual regular optimization between samples are further combined, the structure consistency is kept in the dimension reduction process, a key information structure is kept while the compression ratio is guaranteed, and the high-fidelity and low-redundancy entropy coding compression effect is achieved.

The invention provides a lossless compression method and decompression method for power grid waveform data, equipment, a medium and a program product, and the method comprises the steps: carrying out the periodic feature analysis of the original sampling data of a power grid waveform, so as to determine the periodic structure of the original sampling data, and carrying out the segmentation processing of the original sampling data based on the periodic structure; for the sampling data in each segment, extracting differential information between adjacent sampling points to form a differential data sequence; dynamically determining a corresponding quantization parameter based on the statistical distribution characteristic of the differential data sequence, and performing adaptive quantization processing on the differential data sequence to generate quantized differential data; and entropy coding processing is carried out on the quantized differential data to generate compressed data used for representing the original sampling data, and characteristic parameters used for reconstructing the original sampling data are stored in the compressed data in an associated mode. Unification of high compression ratio, low power consumption and strict lossless reconstruction is realized, and the application requirements of long-term monitoring and remote transmission of the edge side can be met.

The invention discloses a point cloud compression method based on multi-dimensional feature fusion, electronic equipment and a medium, and aims to effectively reduce the compression bit rate while maintaining the precision of a space structure. According to the method, space, channel and topology redundant information is fused under a unified framework for the first time, and the method is suitable for efficient point cloud compression of various three-dimensional scenes such as automatic driving and virtual reality. Meanwhile, a local graph convolution Mama module is introduced, and the limitation that a point cloud topological structure is difficult to capture in a traditional method is effectively solved. Then, the octree nodes are divided into two dimensions of space and channel through a space-channel coupling grouping module, so that gradual feature fusion is realized, and the context information prediction precision is improved; experimental results show that the compression efficiency is remarkably improved on a plurality of radar and voxel human body model data sets.