240 results about "Data compaction" patented technology

The invention provides an attachment packing and unpacking method, system and device based on a compression algorithm and a medium, and belongs to the technical field of file compression. The method comprises the steps that all attachment files needing to be packaged are obtained by traversing a specified directory, screening and filtering are conducted according to file attributes, and a renamed file list and a path mapping table are generated; according to the renamed file list, using a Brotli algorithm to compress each attachment file based on a preset compression level, and packaging the compressed data into a packaged file containing metadata, compressed data and verification information; when the packed file needs to be unpacked, the legality of the packed file is confirmed through magic number verification and index analysis, file data are recovered through Brotli streaming decompression, and the original directory structure of the file is restored. According to the method, efficient compression and streaming decompression are achieved through the Brotli algorithm, and the attachment storage efficiency, the transmission speed and the data integrity are remarkably improved in combination with the intelligent screening and directory structure reduction technology.

The invention provides an industrial safety risk monitoring and early warning system and method for an autonomous planning intelligent agent, relates to the technical field of industrial safety production, and solves various limitation problems still existing in an existing safety risk monitoring scheme. In the system, an environment sensing layer is used for collecting multi-source sensing data in an industrial production environment in real time; the edge calculation layer is used for carrying out primary processing on the multi-source sensing data and executing data compression and key information extraction so as to generate key analysis data; the cloud analysis layer is used for performing deep intelligent analysis on the key analysis data, executing security risk assessment and risk diffusion prediction, and dynamically generating and adjusting an early warning decision through an autonomous planning agent; and the action execution layer is used for executing a corresponding safety action instruction according to the early warning decision and feeding back an execution result to the cloud analysis layer to realize system optimization. According to the invention, the risk early warning accuracy and the emergency disposal timeliness can be improved finally.

The invention relates to the technical field of coal mine transportation system state monitoring, in particular to a cloud side-end collaborative coal mine transportation system state monitoring method and a cloud side-end collaborative coal mine transportation system state monitoring model. In order to solve the problems of difficult multi-modal data fusion, poor dynamic response and insufficient model generalization of a coal mine transportation system, the invention provides a new cloud side-end collaborative coal mine transportation system state monitoring method. Comprising the following steps of multi-modal data acquisition, multi-modal data synchronous transmission, edge computing node data compression, lightweight sensitive feature extraction and transmission, federal learning cross-mine data collaboration, cloud teacher large model construction, knowledge distillation driven student model construction and diagnosis early warning. According to the method, knowledge distillation and federal learning are fused, so that real-time monitoring, accurate diagnosis and efficient maintenance of the coal mine transportation system are realized.

The invention provides an information processing method and system applied to all-flash file storage, and the method comprises the steps: firstly obtaining storage request information received by a target storage node, including a to-be-processed data block set and an expected storage attribute set, thereby describing the configuration requirements of storage capacity distribution, access path optimization and data compression level; extracting multi-dimensional features of a to-be-processed data block set to generate a data block feature vector, encoding attributes of an expected storage attribute set to generate a storage attribute feature vector, inputting the data block feature vector and the storage attribute feature vector into a pre-trained storage strategy generation model for dynamic weight fusion, and outputting a target storage strategy parameter set meeting configuration requirement constraint conditions; finally, relevant parameters are generated, the all-flash storage system is driven to execute distributed storage operation on the data block set, and efficient and intelligent all-flash storage information processing is achieved.

The invention discloses a virtual power plant operation management and control method based on a block chain, original power data are compressed into Hash fingerprints to be chained through a collaborative architecture of an edge computing layer and an AI pre-decision module, the block chain network load is effectively reduced, and a dynamic elastic PBFT consensus mechanism adjusts the scale of verification nodes in real time based on transaction types, so that the efficiency is improved. And compared with a traditional fixed node consensus mechanism, the network communication overhead is reduced. And a hash value asynchronous pre-verification technology is adopted, so that the interaction rate of the multi-dimensional power data is improved, and the efficiency is improved compared with that of a conventional data uplink mode. Through the decoupling design of the intelligent contract execution flow and the consensus verification flow and in combination with the pre-decision data cache queue, the resource scheduling instruction issuing time delay is reduced. The block chain transaction throughput is improved and is effectively improved compared with a standard alliance chain architecture, and the technical problems that the consensus mechanism processing transaction speed of an existing block chain is limited and the high-frequency real-time data interaction requirement of a virtual power plant is difficult to meet are successfully solved.

A computer system for compacting video data. The system acquires a video stream, reduces redundancy through pre-processing, and analyzes the stream to identify patterns and irregularities. It detects spatial or temporal anomalies in the video and produces three outputs: a conditioned video stream based on statistical analysis, an error stream reflecting adjustments made during conditioning, and an anomaly meta-stream containing metadata about detected anomalies. The system communicates with one or more remote systems to synchronize and negotiate a compatible compression codebook, optionally exchanging compact updates that represent differences between local and remote codebooks. The conditioned video stream is then compressed using the agreed codebook. The system outputs a compacted representation of the video that includes the compressed stream, the error stream, and the anomaly metadata, supporting efficient storage or transmission while maintaining the ability to detect, trace, and reconstruct key information within the video.

The invention provides an edge computing remote medical task offloading method based on an improved PPO algorithm, and the method comprises the steps: dynamically selecting an optimal offloading mode through a PPO-MED intelligent decision module according to the characteristics of a medical task and a system resource state, and enabling the offloading mode to comprise a local computing offloading mode (LCO), an edge server offloading mode (ESO) and an edge intelligent preprocessing offloading mode (EIPO); a PPO-MED algorithm models a medical task unloading problem into a Markov decision process, and an unloading strategy is optimized through deep reinforcement learning. The improved PPO-MED algorithm realizes intelligent decision, can dynamically select the optimal unloading strategy according to the real-time system state, task urgency and resource availability, significantly improves the adaptive capacity and service quality of the system in a complex environment, compresses the original medical data into a plurality of orders of magnitude, and improves the reliability of the system. On the premise that the diagnosis accuracy is guaranteed, the transmission delay is remarkably reduced, and the method is particularly suitable for remote diagnosis application of basic medical institutions under the limited bandwidth condition.

A system and method for data compaction with adaptive codebook statistical estimates. Training data sets determine sourceblock frequencies while a dynamic mismatch probability system continuously refines estimates based on observed patterns. Context-aware handling selects appropriate secondary encoding methods for different data types (text, binary, image, executable). Machine learning models predict optimal mismatch probabilities from extracted features. Edge-optimized training enables codebook development on resource-constrained devices with intelligent resource management. Differential updates transmit only changes between codebook versions, minimizing bandwidth usage. Federated learning enables multiple devices to contribute to shared codebooks while maintaining data privacy. A secure synchronization protocol with authentication and verification ensures codebook consistency. The distributed maintenance method provides systematic validation, conflict resolution, and optimization across device networks, enabling efficient encoding across heterogeneous systems.

The invention discloses a file system processing method, electronic equipment and a medium, and relates to the technical field of storage. In the method, data and metadata of a file are compressed. After the mirror image file of the file system is generated according to the super block, the data block and the compressed data, the format of the mirror image disk is the super block + metadata obtained by compressing the metadata of the plurality of files + data obtained by compressing the metadata of the plurality of files. By reorganizing the disk format of the metadata and compressing the metadata into one or more data blocks, the storage space occupied by the file system is effectively reduced; secondly, the metadata of the multiple files are compressed after being integrated, so that when the metadata of the file system is read, the compressed metadata of the multiple files are decompressed in a unified mode, and the decompressing frequency and the IO frequency are reduced; and thirdly, when the metadata of the file is read, the address of the metadata can be directly obtained according to the address information recorded in the super block, so that the time complexity of reading the metadata is reduced.

The invention relates to the technical field of chemical production monitoring, and discloses a cloud computing-based chemical production monitoring system, which is characterized in that an on-site data acquisition module is used for acquiring chemical production data in real time and carrying out filtering and denoising processing to obtain preprocessed data; the network transmission module is used for encrypting the transmitted data and reducing the data transmission quantity by adopting a data compression technology; the cloud computing platform module is used for dispersedly storing the data on a plurality of storage nodes by utilizing a distributed storage technology, classifying the data, analyzing the data in real time by adopting a streaming computing technology, detecting the abnormal condition of the data, and immediately triggering an alarm mechanism when the abnormal condition exceeds a preset safety threshold value; the monitoring application module is used for displaying various parameters and equipment operation states, and providing decision suggestions for production optimization in combination with a chemical production process and equipment; the production efficiency can be improved, the production cost can be reduced, the product quality can be improved, and the market competitiveness of enterprises can be enhanced.

A system and method for data compaction and encryption of anonymized data records. A dataset may be pre-processed by dividing into sourceblocks at reasonable intervals and tallying each sourceblock's frequency, creating a tally record of tokens and count values. This tally record may then be anonymized and transmitted to a data deconstruction engine which combined with a library manager creates a codebook and performs optimization techniques on the codebook. The data deconstruction engine and library manager may be distributed across multiple nodes or devices. The received anonymized tally record may be parsed into individual tokens by identifying the tokens with the highest count value. The tokens may then be sent descending order of count value to the library manger where each token may be assigned a codeword. A half-backed codebook is then created using the tokens and each token's unique codeword, before sending the half-backed codebook to a system user.

The invention discloses a complex environment high-precision positioning system based on multi-signal-source cooperation, and belongs to the technical field of cooperative positioning, and the system specifically comprises the steps: receiving satellite, cellular base station and wireless access point signals based on a multi-source signal extraction module, and extracting carrier phase, propagation time delay and multipath reflection intensity distribution; the reflection weight generation module is used for identifying environmental characteristics according to multipath reflection intensity and generating a fusion weight set containing satellite phase dominant factors and the like; the measurement value constraint conversion module converts the measurement value into three types of distance constraint quantities; the constraint fusion field construction module carries out nonlinear superposition on the three types of constraint quantities according to weights to generate a three-dimensional probability density field; the elevation coordinate resolving module fuses the air pressure data compression resolving space, and positioning coordinates are obtained through iterative convergence; according to the invention, through cooperation of multi-source signals, the positioning precision and reliability in a complex environment are effectively improved.

The invention discloses a digital asset processing method based on a meta universe cultural product. The method comprises the following steps: creating digital assets corresponding to the cultural product; performing right confirmation and authentication processing on the digital assets; carrying out value evaluation on the right-confirmed digital assets; the digital assets are effectively stored and managed; according to the method, the created digital assets are converted into the format suitable for being stored and transmitted on the meta-universe platform, transmission and storage of the digital assets are facilitated, the data compression technology is adopted to reduce the size of the digital assets, the transmission efficiency is improved, and the data transmission efficiency is improved. Meanwhile, the digital assets are dispersedly stored on a plurality of storage nodes by adopting a distributed storage technology, so that the storage nodes of corresponding categories can be conveniently and accurately established according to the types of the digital assets, different types of digital assets can be stored in the storage nodes of the corresponding categories, and subsequent query and acquisition of the digital assets are facilitated.

The invention discloses an NVMe storage array virtualization device based on an FPGA (Field Programmable Gate Array), which is characterized in that single I / O (Input / Output) virtualization (SR-IOV) is realized in the FPGA, and a plurality of NVMe virtual functions which can be directly accessed by a virtual machine are externally provided; mapping from a virtual volume logic address to a physical storage address is completed in hardware, striping management and RAID verification calculation are carried out on a plurality of NVMe solid state disks at the rear end, and storage resource pooling is achieved; the partial reconfigurable characteristic of the FPGA is utilized, I / O scheduling, data compression and prefetching strategies are dynamically adjusted, and the adaptability and performance efficiency of the system are improved. Hardware unloading of the whole I / O path is completed in the FPGA, intervention of a host CPU is not needed, delay is remarkably reduced, the throughput capacity is improved, service quality isolation and elastic expansion in a multi-tenant environment are supported, and an effective technical path is provided for constructing a next-generation high-performance and low-delay storage infrastructure.

The invention discloses an electric vehicle transportation state real-time monitoring method based on multi-sensor fusion, and belongs to the field of electric vehicle transportation state real-time monitoring. A comprehensive state index is obtained through integration of the multi-source data fusion model; acquiring a data transmission priority in the comprehensive state index, and if the data transmission priority is higher than a communication bandwidth threshold, allocating an additional transmission time sequence to obtain a dynamic communication scheduling sequence; carrying out resource allocation on the vehicle-mounted communication system according to the dynamic communication scheduling sequence, and judging whether key safety data such as battery overheating early warning is in the first position of the sequence or not so as to determine a real-time transmission path; and monitoring a communication congestion index in the real-time transmission path, and if the communication congestion index exceeds a threshold value, activating a standby data compression mechanism to obtain a reliable transmission data packet, and sending the reliable transmission data packet to a monitoring center.

Codebook data compaction using hierarchical behavior codebooks to dynamically manage encoding policies and improve entropy encoding methods. A hierarchy of behavior codebooks is maintained, allowing child codebooks to inherit properties from parent codebooks. Behavior codebooks define rules, limitations, and policies for encoding, including sourceblock prioritization and recursive compaction. Conflicts between inherited behaviors are resolved across the hierarchy, and updates to parent codebooks propagate to child codebooks. A behavior codebook manager oversees inheritance validation, policy enforcement, and version control. Data is encoded and decoded using the codebook and the resolved behavior codebooks, ensuring efficient and adaptive data compaction.

The invention relates to the technical field of data storage, and discloses a data storage method, device and equipment, a storage medium and a computer program product, and the method comprises the steps: obtaining power plant data generated in a current time period of a power plant, and initializing a storage node and survival time corresponding to the storage node based on a data compression result of a last time period; determining a data level of each data object in the power plant data; storing the data objects into storage nodes according to the data levels, and activating survival time to obtain an initial storage set; and when the survival time starts to be shortened, adjusting the initial storage set based on the monitored change condition of the survival time to obtain a target storage set. The initial storage set is adjusted according to the change condition of the survival time, the storage structure is dynamically optimized according to the timeliness and importance change of the data in real time, it is guaranteed that important data are always located at proper storage nodes, waste of storage resources is avoided, and the overall performance of a storage system and the reasonability of data management are improved.

The invention discloses an unmanned aerial vehicle group collaborative path planning method and system, a product and a medium, and relates to the field of non-electrical variable control or adjustment systems.The method comprises the steps that environmental data such as terrain height, obstacle distribution and dynamic targets in an unmanned aerial vehicle detection range are collected and converted into a grid unit space occupation state, a search space is divided into three-dimensional grids, and the three-dimensional grids are divided into three-dimensional data; and calculating an unexplored probability value of each unit, extracting time dimension change data of the value and gradient information between the units, identifying an environment dynamic change area, determining boundary features and generating path node data. Node data is compressed into lightweight path information containing coordinates, priority and time sequence constraints, and a track sequence is generated and distributed by combining the position of the unmanned aerial vehicle. By implementing the method, the detection efficiency of the unmanned aerial vehicle can be improved.

Provided is a data compaction method that includes: determining, by an electronic device, a plurality of Sorted String Table (SSTable) files on which a compaction operation is to be performed among SSTable files stored in a storage device connected to the electronic device, the data in the SSTable files stored in the storage device is stored based on a Log Structured Merge Tree (LSM-Tree) mechanism; based on receiving the plurality of SSTable files from the storage device, performing, by the electronic device, merge sorting on data in data blocks of the plurality of SSTable files based on indexes of the data in the data blocks of the plurality of SSTable files, to obtain reconstructed data blocks; and sending, by the electronic device, the reconstructed data blocks to the storage device, the reconstructed data blocks being used by the storage device to obtain reconstructed SSTable files.

The invention discloses a million-account-level distillation mixed inspection method based on SAAS financial software, and the method achieves the cross-enterprise mixed inspection and correction and knowledge closed-loop feedback system through the establishment of a voucher abnormal distillation mechanism, solves the auditing efficiency problem of million-account-level financial vouchers under an SAAS platform, and is suitable for the scenes of account service, finance and tax auditing and the like. At the auditing efficiency level, the to-be-audited data is compressed through an abnormal voucher distillation mechanism, and the exponential-level shortening of the auditing period of millions of enterprises is realized in cooperation with batch processing; in the aspect of auditing quality, industry depth rules cover scenes such as cost allocation of the manufacturing industry and income confirmation of the service industry, and quantitative criteria are provided for differential indexes; the correction efficiency is fundamentally broken through, and a correction result conduction mechanism in the same industry is established for the first time; the knowledge precipitation system realizes automatic conversion from artificial experience to reusable rules, so that finance and tax policy changes can respond within 24 hours.

A system and method for data compaction and encryption of anonymized data records. A dataset may be pre-processed by dividing into sourceblocks at reasonable intervals and tallying each sourceblock's frequency, creating a tally record of tokens and count values. This tally record may then be anonymized and transmitted to a data deconstruction engine which combined with a library manager creates a codebook and performs optimization techniques on the codebook. The data deconstruction engine and library manager may be distributed across multiple nodes or devices. The received anonymized tally record may be parsed into individual tokens by identifying the tokens with the highest count value. The tokens may then be sent descending order of count value to the library manger where each token may be assigned a codeword. A half-backed codebook is then created using the tokens and each token's unique codeword, before sending the half-backed codebook to a system user.

A method includes obtaining, by a first communication apparatus, training data used to train a dictionary used for data compression. The method also includes determining a group of to-be-updated subsets in the dictionary based on the training data. The method further includes determining dictionary update information corresponding to the group of to-be-updated subsets. The method additionally includes sending the dictionary update information.

This specification relates to file format-based transparent encryption tailored for big data. In some aspects, a method includes receiving a write request including a table with one or more columns to be stored in a storage device; compressing table data in a unit of block, wherein each column includes a number of blocks; generating a column key for each column and a block key for each block including sensitive information; encrypting (i) each block including sensitive information with a corresponding block key and (ii) rest of blocks in each column with a corresponding column key; generating wrapped keys for the column keys and block keys and storing the wrapped keys into a key file; and storing the encrypted blocks of each column into a data file in a data folder of the storage device.

A system and method for data compaction with codebook statistical estimates to improve entropy encoding methods to account for, and efficiently handle, previously-unseen data in data to be compacted. Training data sets are analyzed to determine the frequency of occurrence of each sourceblock in the training data sets. A mismatch probability estimate is calculated comprising an estimated frequency at which any given data sourceblock received during encoding will not have a codeword in the codebook. Entropy encoding is used to generate codebooks comprising codewords for data sourceblocks based on the frequency of occurrence of each sourceblock. A “mismatch codeword” is inserted into the codebook based on the mismatch probability estimate to represent those cases when a block of data to be encoded does not have a codeword in the codebook. During encoding, if a mismatch occurs, a secondary encoding process is used to encode the mismatched sourceblock.

The application discloses a kind of low-code application quick construction method, including the following steps: data preparation, application construction, application release, construction is completed, finally generates application module;First, data preparation is carried out, obtains the construction resource data required for application construction, is compressed and uploaded to construction module, then construction module decompresses data and passes through visual construction page, business logic and data structure, to design page design application function in the way of drag and drop.By this method, non-professional personnel can also directly build the application module that oneself needs, improve the application building speed and user-friendly.

A combined satellite and mobile weather forecasting system may include a satellite payload having a plurality of sensors configured to capture atmospheric and environmental data including atmospheric radiances and vertical soundings and a receiver for receiving data from other satellites. The satellite payload may also include a processing unit configured to condense the captured atmospheric and environmental data and the received data from other satellites into initialization data to generate a weather forecast. The satellite payload may include a transmission device configured to communicate with a mobile cellular device and to transmit the initialization data. The mobile cellular device may be configured to receive the initialization data from the transmission device of the satellite payload to generate the weather forecast using the initialization data.

The invention relates to the technical field of Internet of Things, and particularly discloses an Internet of Things data acquisition method and system, and the method comprises the steps: constructing a three-stage data acquisition architecture composed of a main control node, edge acquisition nodes and terminal sensing equipment; the terminal sensing equipment dynamically adjusts own data acquisition frequency based on a built-in activeness evaluation module; the edge acquisition node receives the data sent by the terminal sensing device and executes distributed data verification and data compression processing; the main control node schedules the processed data uploaded by the edge acquisition node according to the network load condition and the data priority; and the main control node performs aggregation analysis on the received data, and optimizes the acquisition strategy of the whole network based on the analysis result. The acquisition frequency can be automatically adjusted according to the equipment state and the data change, invalid data acquisition is reduced, and the equipment energy consumption is reduced. According to the distributed data verification mechanism, the integrity and non-tampering property of the data are ensured, and the credibility of the data is improved.

The invention relates to a model reduced-order acceleration method for HIL simulation of an automobile thermal system model, and belongs to the technical field of automobile thermal system modeling and simulation, and the method comprises the following steps: building a thermal system model: building the thermal system model, generating an input-output data set covering all working conditions, and setting a control equation of the thermal system model; performing principal component analysis (PCA) data preprocessing: compressing high-dimensional space-time data into a low-dimensional feature space, and reserving a dominant dynamic mode; neural state space modeling: designing a nonlinear dynamic system to describe state evolution after dimensionality reduction, establishing a low-dimensional nonlinear dynamic model, and performing network training and regularization; and model integration simulation: integrating the heat exchanger model after order reduction into a complete thermal system model, and performing real-time simulation optimization.

A server apparatus disclosed in the present document may comprise a communication circuit, a memory, a time series database, and at least one processor. The memory may store instructions that cause the server apparatus to: store, in a first table, a label included in time series data and a first hash value corresponding to the label; receive the time series data from an external device through the communication circuit; generate a second hash value by hashing the label of the received time series data; check whether the second hash value is included in the first table; if the second hash value is included in the first table, store, in a second table, the time series data from which the label has been removed and the second hash value; check data prior to a reference time, from among the time series data stored in the second table; and compress a designated data range from among the data prior to the reference time and store same in a third table.

A system and method for lossy precompression for data compaction using automated model monitoring and training, wherein statistical analyses of test datasets are used to determine if the probability distribution of two datasets are within a pre-determined range, and responsive to that determination new encoding and decoding algorithms may be retrained in order to produce new data sourceblocks, and pre-compression of data prior to processing and statistical analysis allows for the compaction of already compressed data into highly dense formats. The new data sourceblocks may then be processed and assigned new codewords which are compiled into an updated codebook which may be distributed back to encoding and decoding systems and devices.