301 results about "Data compaction" patented technology

The invention discloses an efficient auditing method and system based on an edge computing technology, and relates to the technical field of auditing methods and systems.The efficient auditing method comprises the following steps that S1, data collection and preprocessing are conducted, specifically, business data are collected in real time through various sensors and interfaces at edge nodes, and preprocessing is conducted; s2, edge side data analysis: performing real-time analysis on the preprocessed data by utilizing edge computing resources; s3, feature extraction and model training; s4, constructing a risk assessment and early warning model based on edge analysis and a model training result; s5, edge and cloud collaborative auditing; the edge node compresses and encrypts the processing result and the key feature data and then uploads the data to the cloud; and S6, visualizing an audit result and generating a report. According to the method, data acquisition and preprocessing, analysis and model training are realized through edge calculation, the auditing efficiency and accuracy are improved, a risk assessment and early warning model is constructed, the edge and the cloud are coordinated, and comprehensive, timely and reliable auditing support is provided for enterprises.

The invention belongs to the technical field of data synchronization, and particularly relates to a real-time data synchronization method for cloud computing heterogeneous resources. In order to solve the problem of data synchronization among heterogeneous resources, the method comprises the following steps: aggregating a plurality of files in batches to generate a packaged file; performing metadata compression processing on the generated packaged file, and eliminating repeated data through a deduplication algorithm; dynamically selecting a lightweight data transmission protocol; establishing a multi-dimensional index structure; only retaining the file of the latest version through a version control mechanism; dynamically adjusting the allocation of synchronization tasks based on the load condition of available resources in the cloud computing environment; and rescheduling the uncompleted synchronization task. Through a multi-dimensional index structure, version control, self-adaptive transmission scheduling and a dynamic resource allocation mechanism, efficient and stable file synchronization among heterogeneous resources is realized, repeated transmission and storage waste are reduced, and real-time performance, reliability and resource utilization efficiency of synchronization are improved.

The invention provides an attachment packing and unpacking method, system and device based on a compression algorithm and a medium, and belongs to the technical field of file compression. The method comprises the steps that all attachment files needing to be packaged are obtained by traversing a specified directory, screening and filtering are conducted according to file attributes, and a renamed file list and a path mapping table are generated; according to the renamed file list, using a Brotli algorithm to compress each attachment file based on a preset compression level, and packaging the compressed data into a packaged file containing metadata, compressed data and verification information; when the packed file needs to be unpacked, the legality of the packed file is confirmed through magic number verification and index analysis, file data are recovered through Brotli streaming decompression, and the original directory structure of the file is restored. According to the method, efficient compression and streaming decompression are achieved through the Brotli algorithm, and the attachment storage efficiency, the transmission speed and the data integrity are remarkably improved in combination with the intelligent screening and directory structure reduction technology.

The invention provides an industrial safety risk monitoring and early warning system and method for an autonomous planning intelligent agent, relates to the technical field of industrial safety production, and solves various limitation problems still existing in an existing safety risk monitoring scheme. In the system, an environment sensing layer is used for collecting multi-source sensing data in an industrial production environment in real time; the edge calculation layer is used for carrying out primary processing on the multi-source sensing data and executing data compression and key information extraction so as to generate key analysis data; the cloud analysis layer is used for performing deep intelligent analysis on the key analysis data, executing security risk assessment and risk diffusion prediction, and dynamically generating and adjusting an early warning decision through an autonomous planning agent; and the action execution layer is used for executing a corresponding safety action instruction according to the early warning decision and feeding back an execution result to the cloud analysis layer to realize system optimization. According to the invention, the risk early warning accuracy and the emergency disposal timeliness can be improved finally.

The invention relates to the technical field of coal mine transportation system state monitoring, in particular to a cloud side-end collaborative coal mine transportation system state monitoring method and a cloud side-end collaborative coal mine transportation system state monitoring model. In order to solve the problems of difficult multi-modal data fusion, poor dynamic response and insufficient model generalization of a coal mine transportation system, the invention provides a new cloud side-end collaborative coal mine transportation system state monitoring method. Comprising the following steps of multi-modal data acquisition, multi-modal data synchronous transmission, edge computing node data compression, lightweight sensitive feature extraction and transmission, federal learning cross-mine data collaboration, cloud teacher large model construction, knowledge distillation driven student model construction and diagnosis early warning. According to the method, knowledge distillation and federal learning are fused, so that real-time monitoring, accurate diagnosis and efficient maintenance of the coal mine transportation system are realized.

The invention provides an information processing method and system applied to all-flash file storage, and the method comprises the steps: firstly obtaining storage request information received by a target storage node, including a to-be-processed data block set and an expected storage attribute set, thereby describing the configuration requirements of storage capacity distribution, access path optimization and data compression level; extracting multi-dimensional features of a to-be-processed data block set to generate a data block feature vector, encoding attributes of an expected storage attribute set to generate a storage attribute feature vector, inputting the data block feature vector and the storage attribute feature vector into a pre-trained storage strategy generation model for dynamic weight fusion, and outputting a target storage strategy parameter set meeting configuration requirement constraint conditions; finally, relevant parameters are generated, the all-flash storage system is driven to execute distributed storage operation on the data block set, and efficient and intelligent all-flash storage information processing is achieved.

The invention discloses a virtual power plant operation management and control method based on a block chain, original power data are compressed into Hash fingerprints to be chained through a collaborative architecture of an edge computing layer and an AI pre-decision module, the block chain network load is effectively reduced, and a dynamic elastic PBFT consensus mechanism adjusts the scale of verification nodes in real time based on transaction types, so that the efficiency is improved. And compared with a traditional fixed node consensus mechanism, the network communication overhead is reduced. And a hash value asynchronous pre-verification technology is adopted, so that the interaction rate of the multi-dimensional power data is improved, and the efficiency is improved compared with that of a conventional data uplink mode. Through the decoupling design of the intelligent contract execution flow and the consensus verification flow and in combination with the pre-decision data cache queue, the resource scheduling instruction issuing time delay is reduced. The block chain transaction throughput is improved and is effectively improved compared with a standard alliance chain architecture, and the technical problems that the consensus mechanism processing transaction speed of an existing block chain is limited and the high-frequency real-time data interaction requirement of a virtual power plant is difficult to meet are successfully solved.

A computer system for compacting video data. The system acquires a video stream, reduces redundancy through pre-processing, and analyzes the stream to identify patterns and irregularities. It detects spatial or temporal anomalies in the video and produces three outputs: a conditioned video stream based on statistical analysis, an error stream reflecting adjustments made during conditioning, and an anomaly meta-stream containing metadata about detected anomalies. The system communicates with one or more remote systems to synchronize and negotiate a compatible compression codebook, optionally exchanging compact updates that represent differences between local and remote codebooks. The conditioned video stream is then compressed using the agreed codebook. The system outputs a compacted representation of the video that includes the compressed stream, the error stream, and the anomaly metadata, supporting efficient storage or transmission while maintaining the ability to detect, trace, and reconstruct key information within the video.

The invention provides an edge computing remote medical task offloading method based on an improved PPO algorithm, and the method comprises the steps: dynamically selecting an optimal offloading mode through a PPO-MED intelligent decision module according to the characteristics of a medical task and a system resource state, and enabling the offloading mode to comprise a local computing offloading mode (LCO), an edge server offloading mode (ESO) and an edge intelligent preprocessing offloading mode (EIPO); a PPO-MED algorithm models a medical task unloading problem into a Markov decision process, and an unloading strategy is optimized through deep reinforcement learning. The improved PPO-MED algorithm realizes intelligent decision, can dynamically select the optimal unloading strategy according to the real-time system state, task urgency and resource availability, significantly improves the adaptive capacity and service quality of the system in a complex environment, compresses the original medical data into a plurality of orders of magnitude, and improves the reliability of the system. On the premise that the diagnosis accuracy is guaranteed, the transmission delay is remarkably reduced, and the method is particularly suitable for remote diagnosis application of basic medical institutions under the limited bandwidth condition.

The present invention relates to the field of smart grid technology, specifically a distribution terminal communication network system based on WAPI, the system includes a distribution data collection module, a distribution data compression module, a data encryption processing module, a network parameter adjustment module, a distribution anomaly identification module and a distribution scheduling adjustment module. In the present invention, by introducing real-time data processing and encryption technology based on WAPI, the security of data and the efficiency of the network are improved, the data encryption processing ensures the security of information during transmission, enhances the system's control ability over power network operation and monitoring, and adjusts the sending frequency of data packets according to the network status, dynamically optimizes network parameters, improves the network's adaptability and stability, and implements anomaly identification and fault response to ensure the continuity and safety of power grid operation. By optimizing the data processing process and strengthening safety measures, the overall performance and reliability of the power system are significantly improved.

A system and method for data compaction with adaptive codebook statistical estimates. Training data sets determine sourceblock frequencies while a dynamic mismatch probability system continuously refines estimates based on observed patterns. Context-aware handling selects appropriate secondary encoding methods for different data types (text, binary, image, executable). Machine learning models predict optimal mismatch probabilities from extracted features. Edge-optimized training enables codebook development on resource-constrained devices with intelligent resource management. Differential updates transmit only changes between codebook versions, minimizing bandwidth usage. Federated learning enables multiple devices to contribute to shared codebooks while maintaining data privacy. A secure synchronization protocol with authentication and verification ensures codebook consistency. The distributed maintenance method provides systematic validation, conflict resolution, and optimization across device networks, enabling efficient encoding across heterogeneous systems.

The invention discloses a file system processing method, electronic equipment and a medium, and relates to the technical field of storage. In the method, data and metadata of a file are compressed. After the mirror image file of the file system is generated according to the super block, the data block and the compressed data, the format of the mirror image disk is the super block + metadata obtained by compressing the metadata of the plurality of files + data obtained by compressing the metadata of the plurality of files. By reorganizing the disk format of the metadata and compressing the metadata into one or more data blocks, the storage space occupied by the file system is effectively reduced; secondly, the metadata of the multiple files are compressed after being integrated, so that when the metadata of the file system is read, the compressed metadata of the multiple files are decompressed in a unified mode, and the decompressing frequency and the IO frequency are reduced; and thirdly, when the metadata of the file is read, the address of the metadata can be directly obtained according to the address information recorded in the super block, so that the time complexity of reading the metadata is reduced.

The invention relates to the technical field of data processing, in particular to a BIM-based traffic engineering construction progress management method and system, and the method comprises the steps: equally dividing a to-be-analyzed asphalt pavement region into a plurality of block pavement regions; the compaction degree of each block pavement area is obtained; constructing a compactness matrix according to the compactness of all block pavement areas in the to-be-analyzed asphalt pavement area; obtaining a plurality of final similar roadbed matrixes; obtaining the similarity of the compactness in the final similar roadbed matrix; determining a final block of the final similar roadbed matrix according to the similarity; and according to the sizes of the final blocks of the final similar roadbed matrixes, discrete cosine transform compression storage is carried out on all the final similar roadbed matrixes. According to the method, the compaction degree data compression and storage effect of the asphalt pavement area in the BIM-based traffic engineering construction progress process is improved.

The invention relates to the technical field of chemical production monitoring, and discloses a cloud computing-based chemical production monitoring system, which is characterized in that an on-site data acquisition module is used for acquiring chemical production data in real time and carrying out filtering and denoising processing to obtain preprocessed data; the network transmission module is used for encrypting the transmitted data and reducing the data transmission quantity by adopting a data compression technology; the cloud computing platform module is used for dispersedly storing the data on a plurality of storage nodes by utilizing a distributed storage technology, classifying the data, analyzing the data in real time by adopting a streaming computing technology, detecting the abnormal condition of the data, and immediately triggering an alarm mechanism when the abnormal condition exceeds a preset safety threshold value; the monitoring application module is used for displaying various parameters and equipment operation states, and providing decision suggestions for production optimization in combination with a chemical production process and equipment; the production efficiency can be improved, the production cost can be reduced, the product quality can be improved, and the market competitiveness of enterprises can be enhanced.

A system and method for data compaction and encryption of anonymized data records. A dataset may be pre-processed by dividing into sourceblocks at reasonable intervals and tallying each sourceblock's frequency, creating a tally record of tokens and count values. This tally record may then be anonymized and transmitted to a data deconstruction engine which combined with a library manager creates a codebook and performs optimization techniques on the codebook. The data deconstruction engine and library manager may be distributed across multiple nodes or devices. The received anonymized tally record may be parsed into individual tokens by identifying the tokens with the highest count value. The tokens may then be sent descending order of count value to the library manger where each token may be assigned a codeword. A half-backed codebook is then created using the tokens and each token's unique codeword, before sending the half-backed codebook to a system user.

The invention provides a ship magnetic conversion power state remote monitoring system comprising a magnetic conversion power parameter acquisition module which captures magnetic conversion power state parameters of magnetic field change, power output and the like of a ship under different working conditions through a multi-data acquisition device; the remote communication and data transmission module transmits the acquired magnetic conversion power state parameters to a remote control center through a self-adaptive communication protocol; the adaptive communication protocol automatically adjusts the transmission rate and the data compression mode according to the network condition; the optimization algorithm and safety control module operates in a remote control center, and receives and analyzes the transmitted magnetic conversion power state parameters; through a built-in magnetic transformation power optimization algorithm model, the operation state of the ship is evaluated in real time, and potential risks are predicted; when an anomaly is detected, a fluid-actuated safety device is enabled, and related personnel are notified via a remote fault alert system. According to the invention, comprehensive monitoring, efficient transmission and intelligent management of the magnetic conversion power state of the ship are realized.

Codebook data compaction using a universal codebook and mismatch probability estimations to improve entropy encoding methods. Training data sets are analyzed to determine the frequency of occurrence of each sourceblock in the training data sets. A mismatch probability estimate is calculated comprising an estimated frequency at which any given data sourceblock received during encoding will not have a codeword in the codebook. Entropy encoding is used to generate codebooks comprising codewords for data sourceblocks based on the frequency of occurrence of each sourceblock. A "mismatch codeword" is inserted into the codebook based on the mismatch probability estimate to represent those cases when a block of data to be encoded does not have a codeword in the codebook.

The invention discloses a complex environment high-precision positioning system based on multi-signal-source cooperation, and belongs to the technical field of cooperative positioning, and the system specifically comprises the steps: receiving satellite, cellular base station and wireless access point signals based on a multi-source signal extraction module, and extracting carrier phase, propagation time delay and multipath reflection intensity distribution; the reflection weight generation module is used for identifying environmental characteristics according to multipath reflection intensity and generating a fusion weight set containing satellite phase dominant factors and the like; the measurement value constraint conversion module converts the measurement value into three types of distance constraint quantities; the constraint fusion field construction module carries out nonlinear superposition on the three types of constraint quantities according to weights to generate a three-dimensional probability density field; the elevation coordinate resolving module fuses the air pressure data compression resolving space, and positioning coordinates are obtained through iterative convergence; according to the invention, through cooperation of multi-source signals, the positioning precision and reliability in a complex environment are effectively improved.

The invention discloses a digital asset processing method based on a meta universe cultural product. The method comprises the following steps: creating digital assets corresponding to the cultural product; performing right confirmation and authentication processing on the digital assets; carrying out value evaluation on the right-confirmed digital assets; the digital assets are effectively stored and managed; according to the method, the created digital assets are converted into the format suitable for being stored and transmitted on the meta-universe platform, transmission and storage of the digital assets are facilitated, the data compression technology is adopted to reduce the size of the digital assets, the transmission efficiency is improved, and the data transmission efficiency is improved. Meanwhile, the digital assets are dispersedly stored on a plurality of storage nodes by adopting a distributed storage technology, so that the storage nodes of corresponding categories can be conveniently and accurately established according to the types of the digital assets, different types of digital assets can be stored in the storage nodes of the corresponding categories, and subsequent query and acquisition of the digital assets are facilitated.

The invention discloses an NVMe storage array virtualization device based on an FPGA (Field Programmable Gate Array), which is characterized in that single I / O (Input / Output) virtualization (SR-IOV) is realized in the FPGA, and a plurality of NVMe virtual functions which can be directly accessed by a virtual machine are externally provided; mapping from a virtual volume logic address to a physical storage address is completed in hardware, striping management and RAID verification calculation are carried out on a plurality of NVMe solid state disks at the rear end, and storage resource pooling is achieved; the partial reconfigurable characteristic of the FPGA is utilized, I / O scheduling, data compression and prefetching strategies are dynamically adjusted, and the adaptability and performance efficiency of the system are improved. Hardware unloading of the whole I / O path is completed in the FPGA, intervention of a host CPU is not needed, delay is remarkably reduced, the throughput capacity is improved, service quality isolation and elastic expansion in a multi-tenant environment are supported, and an effective technical path is provided for constructing a next-generation high-performance and low-delay storage infrastructure.

The invention belongs to the technical field of automobile data storage, and provides a data storage method and system based on field-level association storage, and the method comprises the steps: building a storage structure for data to be stored based on fields in a distributed database, and associating the fields; pre-processing the associated storage data, and performing timing data compression and duplicate removal on the pre-processed storage data; and optimizing a query path for the preprocessed storage data based on index establishment and a caching technology. According to the method, the data structure is remarkably simplified by adopting a field-level associated storage mode. According to the method, a lengthy and complex table structure in a traditional data model is disassembled, fields are used as basic units for storage, and an efficient association index is established according to the logic relation between the fields. The data storage efficiency is greatly improved, the data writing and reading speed is remarkably increased, and a solid foundation is laid for subsequent data processing and analysis.

The invention discloses an electric vehicle transportation state real-time monitoring method based on multi-sensor fusion, and belongs to the field of electric vehicle transportation state real-time monitoring. A comprehensive state index is obtained through integration of the multi-source data fusion model; acquiring a data transmission priority in the comprehensive state index, and if the data transmission priority is higher than a communication bandwidth threshold, allocating an additional transmission time sequence to obtain a dynamic communication scheduling sequence; carrying out resource allocation on the vehicle-mounted communication system according to the dynamic communication scheduling sequence, and judging whether key safety data such as battery overheating early warning is in the first position of the sequence or not so as to determine a real-time transmission path; and monitoring a communication congestion index in the real-time transmission path, and if the communication congestion index exceeds a threshold value, activating a standby data compression mechanism to obtain a reliable transmission data packet, and sending the reliable transmission data packet to a monitoring center.

The invention discloses a logistics block chain data storage method and system. The method comprises the following steps: (1) data collection; (2) data encryption; (3) block construction; (4) a consensus mechanism; (5) data storage; and (6) data access and query. According to the method, a data index and retrieval mechanism and a query algorithm and logic are optimized, and an efficient query interface is provided, so that the data access efficiency and speed can be remarkably improved. According to the technical scheme, a user can obtain needed data more quickly, the response speed and efficiency of the whole service process are improved, measures such as data encryption, access control and identity verification are enhanced, and the safety of the data in the storage, transmission and access processes can be ensured; therefore, data leakage, tampering and unauthorized access can be prevented, benefits of enterprises and users are protected, occupation of storage space can be reduced through technical means of data compression, duplicate removal, redundancy elimination and the like, and storage cost is reduced.

A system and method for lossy precompression for data compaction using automated model monitoring and training, wherein statistical analyses of test datasets are used to determine if the probability distribution of two datasets are within a pre-determined range, and responsive to that determination new encoding and decoding algorithms may be retrained in order to produce new data sourceblocks, and pre-compression of data prior to processing and statistical analysis allows for the compaction of already compressed data into highly dense formats. The new data sourceblocks may then be processed and assigned new codewords which are compiled into an updated codebook which may be distributed back to encoding and decoding systems and devices.

Codebook data compaction using hierarchical behavior codebooks to dynamically manage encoding policies and improve entropy encoding methods. A hierarchy of behavior codebooks is maintained, allowing child codebooks to inherit properties from parent codebooks. Behavior codebooks define rules, limitations, and policies for encoding, including sourceblock prioritization and recursive compaction. Conflicts between inherited behaviors are resolved across the hierarchy, and updates to parent codebooks propagate to child codebooks. A behavior codebook manager oversees inheritance validation, policy enforcement, and version control. Data is encoded and decoded using the codebook and the resolved behavior codebooks, ensuring efficient and adaptive data compaction.

The invention relates to the technical field of data storage, and discloses a data storage method, device and equipment, a storage medium and a computer program product, and the method comprises the steps: obtaining power plant data generated in a current time period of a power plant, and initializing a storage node and survival time corresponding to the storage node based on a data compression result of a last time period; determining a data level of each data object in the power plant data; storing the data objects into storage nodes according to the data levels, and activating survival time to obtain an initial storage set; and when the survival time starts to be shortened, adjusting the initial storage set based on the monitored change condition of the survival time to obtain a target storage set. The initial storage set is adjusted according to the change condition of the survival time, the storage structure is dynamically optimized according to the timeliness and importance change of the data in real time, it is guaranteed that important data are always located at proper storage nodes, waste of storage resources is avoided, and the overall performance of a storage system and the reasonability of data management are improved.

Provided are a computer program product, system, and method for determining to migrate data between compressed storage devices and non-compressed storage devices in a storage system. The storage system provides compressed storage devices having native data compression and a non-compressed storage devices not configured to have native compression. A determination is made of storage usage factors for the compressed storage devices and the non-compressed storage devices. A determination is made whether to move data between the compressed storage devices and the non-compressed storage devices based on the storage usage factors. Data is moved between the compressed storage devices and the non-compressed storage devices in response to determining to move the data between the compressed storage devices and the non-compressed storage devices.

The invention discloses an unmanned aerial vehicle group collaborative path planning method and system, a product and a medium, and relates to the field of non-electrical variable control or adjustment systems.The method comprises the steps that environmental data such as terrain height, obstacle distribution and dynamic targets in an unmanned aerial vehicle detection range are collected and converted into a grid unit space occupation state, a search space is divided into three-dimensional grids, and the three-dimensional grids are divided into three-dimensional data; and calculating an unexplored probability value of each unit, extracting time dimension change data of the value and gradient information between the units, identifying an environment dynamic change area, determining boundary features and generating path node data. Node data is compressed into lightweight path information containing coordinates, priority and time sequence constraints, and a track sequence is generated and distributed by combining the position of the unmanned aerial vehicle. By implementing the method, the detection efficiency of the unmanned aerial vehicle can be improved.

Provided is a data compaction method that includes: determining, by an electronic device, a plurality of Sorted String Table (SSTable) files on which a compaction operation is to be performed among SSTable files stored in a storage device connected to the electronic device, the data in the SSTable files stored in the storage device is stored based on a Log Structured Merge Tree (LSM-Tree) mechanism; based on receiving the plurality of SSTable files from the storage device, performing, by the electronic device, merge sorting on data in data blocks of the plurality of SSTable files based on indexes of the data in the data blocks of the plurality of SSTable files, to obtain reconstructed data blocks; and sending, by the electronic device, the reconstructed data blocks to the storage device, the reconstructed data blocks being used by the storage device to obtain reconstructed SSTable files.

A system, method, and computer-readable medium for performing a data center management and monitoring operation. The data center management and monitoring operation includes: receiving data center data from a plurality of data center assets within a data center, the data center data comprising data center telemetry data; compressing the data center telemetry data to provide compressed data center data; and, performing a data center analytics reconstruction operation, the data center analytics reconstruction operation reconstructing the compressed data center data to provide decompressed data center telemetry data, the data center analytics reconstruction operation using a gated recurrent unit (GRU) network generator.