349 results about "Compress Algorithm" patented technology

The invention is suitable for the technical field of data asset management, and provides a data asset management system based on a block chain and big data analysis, and the system comprises a first terminal which carries out the storage of the ownership information of data assets through a block chain network, and generates a storage data package; generating an evaluation result based on a preset technical index analysis model; writing the hash value of the evaluation result into transaction data of the main chain of the block chain, and generating an evidence storage voucher matched with the block chain transaction; an interaction data sequence is packaged for identity signature, encryption and compression to generate a target code stream, and the target code stream is sent to the second terminal; the second terminal decrypts the encrypted data in the target code stream by using a private key, decompresses the data by using a compression algorithm matched with the first terminal, and restores the data into an interactive data sequence; verifying the access authority and the transaction condition of the evidence storage data packet; checking the anchoring state of the evidence storage voucher on the main chain of the block chain; and the data transaction is completed, the data asset state report is generated, and the data element marketization configuration efficiency is improved.

The invention relates to the technical field of data processing and communication, in particular to a multi-source heterogeneous data compression and transmission method and system for an intelligent fusion terminal, and the method comprises the steps: collecting multi-source heterogeneous data from different data sources through the intelligent fusion terminal, carrying out the data preprocessing of the collected multi-source heterogeneous data, and obtaining a data preprocessing result; comprising format unification and noise filtering; establishing a data feature analysis model, and performing feature extraction and classification on the preprocessed data; according to data feature extraction and classification results, dynamic weights are allocated to different types of data, and the data are divided into data blocks of different sizes according to the allocated dynamic weights and data features; for different types of data blocks, the priority of the data blocks is set, a proper compression algorithm is selected for compression, and the compressed data is transmitted through a network. The multi-source heterogeneous data is subjected to feature analysis and targeted compression processing, so that network bandwidth occupation and equipment energy consumption are reduced, and data transmission efficiency is improved.

The invention relates to the technical field of data transmission of the Internet of Things, in particular to a self-adaptive data compression and transmission method of a low-power-consumption wide-area Internet of Things. According to the method, a three-layer collaborative architecture comprising an equipment layer, a fog computing layer and a cloud computing layer is constructed, data preprocessing and feature analysis are performed on the equipment layer, and data types, entropy values and repeatability information are extracted; the fog calculation layer selects an optimal compression algorithm based on a multi-dimensional decision engine, and reduces redundancy through spatial-temporal correlation analysis and data aggregation; the cloud computing layer collects compression performance data, adopts reinforcement learning and federal learning to train a global optimization model, and dynamically issues strategy parameters; and the fog node adaptively adjusts a compression strategy in combination with the system state to realize the optimal balance between the compression ratio and the reconstruction precision. The method is suitable for field deployment environments with limited electric quantity and network, has the advantages of low power consumption, high efficiency, strong adaptability and the like, and can be widely applied to Internet of Things scenes such as remote monitoring, smart energy and the like.

The invention relates to a three-dimensional model compression transmission method and system based on dynamic feature perception, and relates to the technical field of three-dimensional live-action modeling, and the method comprises the steps: obtaining the original three-dimensional data of a three-dimensional model, carrying out the multi-modal feature extraction based on a model scene, dividing a key region, a transition region and a non-key region, and generating a three-dimensional model partition map, compressing each region according to the partition map label and a preset compression algorithm to generate a multi-resolution LOD sequence, determining a transmission data hierarchy in combination with a network state and equipment parameters, and finally rendering the model according to the transmission data hierarchy, user behavior information and an environment state to obtain a target three-dimensional model. The technical effects of effectively compressing, transmitting and rendering the three-dimensional model according to various factors such as different area characteristics, network states, equipment parameters and user behaviors of the model and improving the transmission efficiency and the rendering quality are achieved.

The invention discloses a time series data lossless compression method and system based on dynamic context awareness, and relates to the field of data lossless compression, and the method comprises the steps: extracting target data from a device port, and carrying out the preprocessing of the target data through a data processing technology according to the business dimension corresponding to the target data, constructing a structured data set based on a preprocessing result; capturing local time sequence features based on the structured data set, distributing attention weights in combination with a hidden state, generating a feature vector used for sensing a context, and outputting a dimension vector by using a three-layer full-connection network; and according to the dimension vector, analyzing the compression performance score value of each compression algorithm in the resource library, sorting the compression performance score values, and selecting the compression algorithm meeting the target requirement to compress the target data. According to the method, the global periodic features and the local mutation features of the time series data are analyzed through the space-time attention mechanism, and real-time driving algorithm switching is achieved.

The invention discloses a neural radiation field rendering method based on dynamic hash coding, and the method comprises the steps: employing the feature sequence data as the input, calculating the density value and color value of each sampling point through the forward propagation of a neural network, carrying out the volume rendering integral operation according to the ray tracing principle in the direction of a ray, and obtaining the feature sequence data; judging a final color output result of the current pixel point; according to an error value between the color output result and a real image, updating a network parameter weight through a back propagation algorithm, and if the error value is greater than a convergence threshold, continuing to iterate the training process to adjust a feature coding strategy to obtain an optimized neural radiation field model parameter; and after the rendering performance configuration parameters are obtained, optimizing a storage allocation strategy of feature data through a memory pool management mechanism, and if the current memory occupancy rate exceeds a safety threshold, starting a data compression algorithm to reduce the storage space requirement, and obtaining a real-time rendering output result. According to the invention, high-quality real-time rendering of the dynamic scene is realized.

This application discloses a parameter advertisement method and apparatus. An unpacking end advertises a packing end of a feature of a data flow that needs to be packed and an identifier of a compression algorithm, so that the feature of the data flow and the identifier of the compression algorithm are automatically transferred from the unpacking end to the packing end. Therefore, the packing end can determine packing for some data flows and a to-be-used compression algorithm based on the received feature of the data flow and the received identifier of the compression algorithm, so that the packing end can pack a specified data flow by using a specified compression algorithm, and a process in which the feature of the data flow and the identifier of the compression algorithm are manually configured at the packing end is avoided. Therefore, configuration complexity at the packing end is reduced.

The invention relates to the technical field of data compression, in particular to a ZSTD compression algorithm optimization method based on joint coding and ternary decomposition, and the method comprises the following steps: S1, building a joint frequency table based on the joint distribution of the literal length and the matching length of a previous section, setting a frequency deviation criterion, and dynamically adjusting a low-frequency combination proportion; s2, decomposing the offset value structure into a leading segment, a trailing segment and a middle segment, performing bit width-entropy ratio analysis on the middle segment, and selecting a symbol index or bit stream coding; and S3, entropy discriminant coding is performed on the joint symbol and the offset symbol respectively, and a structure position is mapped synchronously to generate a structured compressed output stream. According to the method, collaborative optimization of frequency self-adaption, redundancy minimization and structured mapping is realized through joint symbol dynamic modeling and offset ternary structure entropy discriminant coding, so that the ZSTD compression efficiency and the coding stability are remarkably improved.

The invention discloses a memory management method and related equipment, and relates to the technical field of data storage, the method comprises the following steps: based on access information of memory pages of a target operating system, determining a dynamic popularity score of each memory page; on the basis of the current memory pressure level and the dynamic popularity score of the target operating system, executing a corresponding hierarchical data migration strategy so as to migrate different memory pages to the corresponding main Swap storage space or auxiliary Swap storage space; adjusting a compression algorithm applied to the auxiliary Swap storage space based on the current central processing unit utilization rate of the target operating system; and detecting real-time states of the main Swap storage space, the auxiliary Swap storage space and the physical memory, and adjusting a hierarchical data migration strategy and a compression algorithm based on a detection result. Through dynamic popularity perception, hierarchical data migration, adaptive compression regulation and cross-layer collaborative optimization, the memory utilization rate, the exchange efficiency and the overall system stability can be improved under different system loads.

The invention relates to a simulation result compression method and system based on semantic segmentation, equipment and a medium. The method comprises the following steps: performing feature extraction on a simulation result file to obtain semantic features; generating a segmentation point sequence according to the features by using a semantic segmentation model, and segmenting the file into a plurality of semantic blocks; extracting a compressibility feature vector for each block, and dynamically distributing an optimal compression strategy for the block through a prediction model; compressing each block by using the distributed strategy to generate a compressed data block; and finally, constructing a block sensing index based on the block type and the used strategy, and integrating the index and the data block to generate a compressed file. By adopting the method, the defects of limited compression ratio and incapability of supporting rapid random access caused by lack of semantic understanding of a general compression algorithm can be overcome, and by understanding the internal logic structure of the file and applying differential compression, the rapid retrieval and analysis capability of specific data is kept while the high compression ratio is realized.

The invention provides a multi-modal data enhanced storage system based on a Parquet format. Unified storage and optimization of multi-modal data are achieved through native transformation and expansion of the Parquet format. The system comprises a modal perception format expansion component and a modal optimization coding strategy component, wherein the modal perception format expansion component realizes deep fusion of multi-modal semantics and a storage structure by expanding a Parquet logic type system and constructing a four-layer hierarchical metadata architecture from a file level to a page level; and performing instruction level fusion with a Parquet compression algorithm, executing targeted storage optimization based on semantic descriptors and modal characteristic parameters, and solving the problems of unified storage and efficient management of multi-modal data.

The invention provides a natural resource archive retrieval and utilization method combining a large model and a knowledge graph technology, and relates to the technical field of data processing, the method comprises the following steps: identifying business entities in the field of natural resources and semantic association thereof, and constructing an RDF triple with attribute constraint; performing geometric feature preserving type coordinate simplification on the plot entity vector boundary coordinate point sequence through a Douglas-Peucker compression algorithm to generate a key coordinate point set after noise reduction; on the basis of the key coordinate point set, vector geometric superposition operation is carried out on polygon boundaries of the two plot entities, the area proportion of overlapped regions is calculated according to the intersection-to-union ratio, and when the intersection-to-union ratio exceeds a preset threshold value, entity space alignment is carried out, so that a triple set of space alignment is obtained; and fusing the triple set with the aligned space, and constructing the natural resource domain knowledge graph. According to the invention, redundant information interference is effectively reduced.

The invention discloses a method and a system for identifying an anti-counterfeit label. The method comprises the following steps: acquiring spectral response characteristics of the anti-counterfeit label to be identified through a multiband optical sensor; extracting a dynamic coding area and a base material characteristic area in the spectral response characteristics, wherein the dynamic coding area comprises topological distribution information of a programmable encryption particle array; processing the topological distribution information of the dynamic coding region through a feature compression algorithm to generate a first verification factor, and extracting a spatial frequency feature of the base material feature region as a second verification factor; and uploading the first verification factor and the second verification factor to a cloud verification platform through a secure communication protocol, wherein the cloud verification platform is internally provided with a dynamic spectral feature library and a physical feature association model. According to the invention, through a double-engine architecture of programmable dynamic target and environment adaptive verification, the problems of static replicability and environment vulnerability of physical anti-counterfeiting are solved, and collaborative rise of anti-counterfeiting strength and use reliability is realized.

The invention belongs to the technical field of data security, and particularly relates to a non-intrusive power load identification method and system based on an intelligent fusion terminal, and the method comprises the steps: collecting power load data, and dividing an analysis window based on the power load data; calculating a volatility index corresponding to the analysis window based on the data in the analysis window so as to represent the data complexity in the analysis window; determining the number of iterations for encryption based on the volatility index; and executing a compression algorithm on the data in the analysis window to generate compressed data, and executing multiple rounds of encryption perturbations determined by the number of iterations on the compressed data to generate a dynamically encrypted compressed data stream. According to the method, the privacy protection intensity can be adaptively adjusted according to the dynamic complexity of the data, and the privacy leakage risk caused by the fact that the static encryption intensity cannot adapt to the data dynamics in the prior art is solved.

The invention discloses an eddy current data compression transmission optimization method, relates to the technical field of data compression transmission, and solves the technical problems that a compression algorithm lacks scenario adaptation and data transmission lacks priority and link adaptation. Scene demand-algorithm characteristic accurate matching is achieved, compressed data overall reconstruction errors and defect area reconstruction errors are reduced, delay and compression ratio requirements of different scenes are met, data priorities are judged in a multi-dimensional mode, it is ensured that defect data acquisition resources are inclined, key information is prevented from being lost, and the method is suitable for large-scale popularization and application. A transmission protocol is optimized for a short / medium / long distance scene, the protocol adaptability and the transmission efficiency are improved, a transmission scene-link state-data priority three-dimensional linkage transmission strategy is constructed, the bandwidth occupation proportion is dynamically adjusted, core data are preferentially transmitted when links are congested, and resources are fully utilized when the links are idle.

The invention discloses a vehicle data transmission method and device, electronic equipment and a storage medium. The method is applied to a data sending end, and comprises the following steps: in response to a received data transmission instruction, determining to-be-transmitted data corresponding to the data transmission instruction and a data receiving end; determining a target compression algorithm based on target compression configuration information and the multi-dimensional decision information, the target compression configuration information being used for representing compression configuration information jointly supported by the data sending end and the data receiving end, the multi-dimensional decision information at least comprises two of the following: a network resource parameter, a computing resource parameter and a data attribute parameter of the to-be-transmitted data; compressing the to-be-transmitted data by using the target compression algorithm to obtain processed data; and transmitting the processed data to a data receiving end. According to the invention, the technical problem of low transmission efficiency and adaptability of vehicle data transmission in the prior art is solved.

The invention provides a data compression method, apparatus and device, and a storage medium. The method comprises the steps of obtaining an original data stream transmitted through a serial port; determining the data type of the original data flow according to the original data flow; determining a corresponding target compression algorithm according to the data type; compressing the original data stream according to the target compression algorithm to obtain compressed data; the defects of low efficiency and even data expansion during data abrupt change in a fixed compression mode are effectively overcome, and the data transmission efficiency and the bandwidth utilization rate of serial port communication in a resource limited environment are remarkably improved.

One example method includes at a source, at a source, performing a symbolic aggregation process on a series of raw data generated and / or collected by the source, to create a series of symbols, inputting, by the source, the series of raw data and the series of symbols to a lossy compression algorithm operating at the source, running, at the source, the lossy compression algorithm to obtain a series of raw values, and a sparse series of raw values, and transmitting, by the source to a target, the series of raw values, and the sparse series of raw values.

The invention provides a power plant high-frequency data transmission method based on a lossless compression algorithm. The power plant high-frequency data transmission method comprises the following steps of performing quality preprocessing on original high-frequency data; implementing hierarchical acquisition and periodic packaging according to data real-time requirements; performing differential coding for the data type: performing run-length coding on the switching value, and performing differential coding on the analog quantity; carrying out unified lossless compression on the coded data by using a Zstand-ard algorithm; the compressed data is encrypted and transmitted to an industrial internet platform, challenges confronted by a speed regulation system are handled through the steps, automatic optimization and dynamic adjustment of a data compression algorithm are achieved so as to adapt to the requirements of different application scenes, and the stability and efficiency of power production are improved.

The invention relates to the technical field of edge data processing, and particularly provides an edge data transmission method for an intelligent supply chain platform. Carrying out primary processing on the collected data on edge computing equipment; dynamically distributing data processing tasks according to the emergency degree, processing complexity and network condition factors of the data, determining that one part of the data is processed on edge equipment, and uploading the other part of the data to a cloud for processing; a data transmission protocol is optimized, technologies such as a compression algorithm and differential transmission are adopted, the data volume is reduced, and transmission delay and bandwidth occupation are reduced; further analyzing and processing the data uploaded to the cloud to generate decision support information; according to a processing result and system feedback, a task allocation strategy and data transmission parameters are dynamically adjusted, the overall performance of the system is optimized, and the method has the effects of finding the optimal balance between edge processing and cloud processing and avoiding data loss or untimely processing.

The invention provides a Flink multi-cluster security authentication data synchronization method and system, and relates to the technical field of distributed computing, and the method comprises the steps: establishing a secure connection channel, optimizing an execution plan by adopting an incremental coding compression algorithm, setting an adaptive computing load threshold, dynamically allocating computing resources, and writing state check point information. Data quality evaluation is executed, the evaluation data is written into the distributed account book, the state of the target cluster is recovered, the synchronization task is started, the data synchronization safety and efficiency are improved, the computing resource utilization rate is increased, and the data quality and synchronization consistency are ensured.

The invention discloses an electric energy meter data compression method and system, electronic equipment and a storage medium, and the method comprises the steps: carrying out the first-order differential processing of an electric quantity measurement data sequence, obtaining a residual data sequence, greatly reducing the value domain of the residual data sequence, greatly improving the repetition, and remarkably reducing the statistical entropy; low-entropy and high-redundancy ideal input is created for subsequent coding, and the compression ratio is integrally improved. According to the method, the residual data sequence is subjected to coding processing, positive and negative information is reserved, the problem that complement storage equivalent entropy is not reduced is avoided, bit column recombination processing or byte column recombination processing is performed on the residual coding sequence, and a large number of continuous data streams of local repeated fragments are generated, so that in an electric energy meter environment in which the MCU computing power and the RAM are limited subsequently, the residual data sequence is subjected to bit column recombination processing or byte column recombination processing. The lightweight compression algorithm can also quickly discover and reuse redundancy in an extremely short window, and the overall compression efficiency is remarkably improved.

The application discloses an electric energy meter data compression method and system, an electronic device and a storage medium. The method first performs first-order difference processing on an electric quantity measurement data sequence to obtain a residual data sequence. The value range of the residual data sequence is greatly reduced, the repetition degree is greatly increased, and the statistical entropy is significantly reduced, thus creating an ideal input with low entropy and high redundancy for subsequent coding, and improving the compression ratio as a whole. The residual data sequence is then coded, the positive and negative information is retained, and the problem of equivalent entropy not being reduced due to complement storage is avoided. The residual coded sequence is then subjected to bit column reorganization processing or byte column reorganization processing, and a continuous data stream with a large number of locally repeated segments is generated, so that in the subsequent electric energy meter environment with limited MCU computing power and RAM, a lightweight compression algorithm can also quickly find and reuse redundancy in a very short window, and the overall compression efficiency is significantly improved.

The invention provides a user operation recommendation method under an AI platform, and belongs to the technical field of large language models.According to the method, multi-dimensional operation data of a user are obtained in real time by building a distributed data collection system, and page operation records are processed through a clustering compression algorithm; natural language processing and a sequence identification algorithm are used for extracting semantic features and historical interaction behavior modes of inquiry content, a user operation intention graph is constructed, association strength is calculated through a shortest path algorithm, user intentions are accurately modeled, and a multi-modal fusion recommendation model based on a Transform architecture is established; an online learning system is constructed, user satisfaction is calculated through an intention correlation degree function, model parameters are optimized in real time by adopting a gating weight dynamic adjustment function, recommendation accuracy and response speed are ensured by applying a mode of combining offline pre-calculation and online retrieval optimization, and the technical problem that AI platform user operation recommendation is not accurate enough is solved.

A system and method for zero-knowledge verifiable codebook compression receives an input data stream comprising data blocks and encodes the stream using codebook-based compression algorithms. Concurrently with encoding, the system generates zero-knowledge proofs that cryptographically attest that the encoded representation will decode to data having a specified digest and that policy appendices associated with the codebook were applied during encoding. The system generates codebook commitments comprising cryptographic commitments to codebook contents and policy metadata, then formats output packets containing the encoded representation, zero-knowledge proof, and public inputs including the specified digest and codebook commitment. The zero-knowledge proofs enable verification systems to validate encoding correctness and policy compliance without accessing plaintext content or codebook contents.

The text compression serialization method is applied to computer equipment and comprises the steps that text features of a text are extracted, resource information of the computer equipment is collected, the text features comprise a structured index, a repetition rate and semantic entropy, and the resource information comprises a CPU utilization rate and a remaining memory proportion; screening a first candidate compression algorithm set of the text by using the structured index and the repetition rate, filtering the first candidate compression algorithm set based on the resource information, selecting a second candidate compression algorithm set according to the semantic entropy, and determining a target candidate compression algorithm set through the resource information; based on the text features and the resource information, calculating a score value of each candidate compression algorithm in the target candidate compression algorithm set, and taking the candidate compression algorithm with the highest score value as a target compression algorithm; and compressing and serializing the text by adopting a target compression algorithm to obtain a compressed serialized text. Therefore, the flexibility, the compression effect and the resource utilization rate of text compression serialization can be improved.

The invention discloses a multi-modal corpus storage method and device, computer equipment and a medium, and the method comprises the steps: carrying out the compression processing of text modal data and audio and video modal data of a multi-modal corpus, and employing a suitable compression algorithm for the characteristics of different modal data; similarity calculation is carried out on the text compression features, the audio and video compression features and corresponding modal features of the newly-added multi-modal corpus, semantic redundancy of the newly-added multi-modal corpus and historical compression corpus can be recognized, a fusion compression process is triggered to eliminate cross-modal redundancy, and repeated storage of same semantic information is avoided; corpus features which do not trigger the fusion process are independently stored, the corpus features which trigger the fusion process are fused to generate multi-modal joint features, and the stored data volume is further simplified; the corpus features are migrated to the corresponding data storage layer in combination with the access frequency and the response time, so that the storage cost is reduced.

This application discloses a data compression method and apparatus. The method includes: obtaining m data blocks in a data area of a readable and writable file system; compressing the m data blocks by using a preset compression algorithm, to sequentially obtain n compressed data blocks, where first capacities of the compressed data blocks are the same, and the first capacity represents a quantity of bytes of compressed data that can be included in the compressed data block; and establishing a first index of each of j data blocks corresponding to an ith compressed data block in the n compressed data blocks, and recording a mapping relationship between first indexes and the j data blocks. The first index is used to identify a storage location, in a storage medium, of each data block included in the j data blocks, and attribute information included in each of the j data blocks.

The invention discloses a self-adaptive compression and lossless fast transmission method and system for high-frequency test data. The method comprises the following steps: extracting multi-dimensional dynamic feature vectors from high-frequency test data in real time and marking feature points; based on the features, the network bandwidth and the target compression ratio, dynamically selecting a compression algorithm combination and parameters through a pre-training decision model; performing data framing and typed preprocessing according to the algorithm combination; compression is executed through a multi-stage assembly line, and a correction link based on statistical characteristics is inserted between stages; constructing a three-dimensional transmission state matrix fusing priorities, links and quality scores, dynamically allocating links and adaptively adding forward error correction codes; and after error correction and decompression of a receiving end, realizing lossless verification through multi-dimensional comparison of feature point sequences of the reconstructed data and the original data. According to the method, the whole-course self-adaption of the compression strategy and the transmission scheduling is realized, and the lossless and reliable recovery of the data is ensured while the compression and transmission efficiency is improved.

The invention relates to a configurable decompression circuit supporting COO and Bi tmap compression algorithms, and belongs to the field of integrated circuits. According to the hybrid compression method suitable for the circuit, after sparseness analysis is carried out on weight matrixes of all layers of a neural network, COO compression based on a coordinate type sparse matrix or Bitmap compression based on bitmap masks is selected according to the sparseness characteristics of different layers, so that the storage space is optimized; comprising a control module, a first selector, a second selector, a bitmap description memory, a coordinate index memory, a numerical memory, a converter, a COO decoder and a decompression data storage module, decoding of two compression formats of COO and Bi tmap can be supported, and the operation efficiency and flexibility of a neural network processor are improved. The method effectively reduces the weight storage demand of the neural network, enhances the adaptability of the compression algorithm, and is suitable for hardware implementation of an efficient neural network model.