1484 results about "Storage efficiency" patented technology

The invention provides a meteorological metadata storage method and system based on machine learning, and the method comprises the steps: obtaining a meteorological metadata set of a target region, inputting the meteorological metadata set into a pre-trained time sequence feature extraction model, generating the time sequence distribution features of all meteorological file files, and inputting a spatial topology mapping model, generating spatial correlation topological features of the meteorological elements; based on a cross fusion result of the time sequence distribution features and the space correlation topological features, a hierarchical index structure of meteorological metadata is constructed, a meteorological storage optimization model is called to generate a storage node distribution strategy of a meteorological file, and the meteorological file is stored according to the storage node distribution strategy. And writing each meteorological file file in the meteorological metadata set into the corresponding distributed storage node, and dynamically adjusting the mapping relationship between the physical position and the index of the stored file based on a real-time updating mechanism of the hierarchical index. The meteorological metadata storage efficiency and retrieval accuracy can be improved, and storage resource waste and operation and maintenance cost are reduced.

The invention discloses an optical fiber data storage management system and method based on big data, and relates to the technical field of storage management. According to the optical fiber data storage management system and method based on big data, the spatial-temporal characteristics of the historical access records are extracted through the LSTM network, the data heat value is calculated in combination with the time decay function, and the thermal distribution diagram of the data blocks is generated. And based on the thermal distribution map, constructing a storage resource allocation model by using a deep reinforcement learning algorithm, and dynamically adjusting the optical path priority and copy distribution. A data prefetching strategy is optimized through the spatio-temporal joint index and the vector similarity index, and a high-frequency data cache index distribution diagram is generated. And comprehensively considering the data access delay, the index hit rate and the copy migration frequency, obtaining an optical fiber storage efficiency index, and adjusting a storage strategy according to the index. According to the method, the dynamic scheduling and utilization efficiency of data storage resources is effectively improved, the data access delay is reduced, the copy management is optimized, and the overall performance of a storage system is remarkably improved.

The invention relates to a method convenient for data consanguinity collection and analysis, which comprises the following steps of: obtaining original consanguinity data comprising a task execution log, application metadata and a cross-system dependency relationship, and carrying out standardization processing on the original consanguinity data to generate structured consanguinity information comprising an asset unique identifier, an upstream and downstream association relationship and a data operation type; structured consanguinity information is synchronously written into a graph database and a distributed data warehouse, the graph database stores real-time association topology, the distributed data warehouse stores full-amount historical versions, and a transaction consistency algorithm is adopted to ensure the atomicity of double-write operation, so that the data storage efficiency is improved. Single-asset-level consanguinity tracking is performed based on real-time topology of a graph database, global consanguinity analysis is performed based on batch computing power of a distributed data warehouse, a direct dependence path, a deep association network and a closed-loop link detection result are generated, and a closed-loop management mechanism from data acquisition, analysis to optimization is formed. And the problem that an analysis result is disjointed from an acquisition end in a traditional scheme is solved.

The invention relates to the technical field of information management, and comprises a distribution automation operation and maintenance work log intelligent recording platform based on a CS architecture, and the platform comprises a log field analysis module, a task chain construction module, a task logic verification module, and a log storage tracing module. According to the method, missing and abnormal fields are effectively marked and corrected through meticulous analysis and comparison of uploaded log records, the accuracy and integrity of data are improved, task scheduling is optimized and the operation and maintenance management efficiency is improved through association of device numbers and task types and adjustment of execution time in sorting and grouping of task sequences, and the task scheduling efficiency is improved. Task chains are recombined to solve conflicts and adjust priorities, smooth execution of operation and maintenance tasks is guaranteed, delay is reduced, logic verification enhances task reliability through rule verification and error marking, unassociated tasks are deduced, analyzed and corrected, the exception processing capacity is enhanced, the data tracing and storage efficiency is improved, and the tracing capacity of data management is enhanced.

The embodiment of the invention provides a kafka storage and calculation separated data processing method and device, and the method comprises the steps: constructing a multi-layer data processing architecture which comprises four layers: network communication, task distribution, processing scheduling and storage. Cold and hot data separation is realized through hierarchical design of a log pre-writing area and a main storage area, and intelligent scheduling of system resources is realized in combination with a copy manager, a coordinator and a load predictor. An adaptive compression algorithm and a verification mechanism are adopted to ensure the data transmission efficiency and integrity, a message cache region is created to store hotspot data, and flexible access of cold and hot data is realized based on message sites. According to the method, the defects of the traditional technology in storage and calculation coupling, storage efficiency, load balancing and the like are effectively overcome, and the performance and expandability of a message queue system are remarkably improved.

The invention discloses a hyper-converged server multi-resource integration system and a scheduling method, belongs to the technical field of computer resource management, and aims at solving the problems that a traditional hyper-converged server is dispersed in resource scheduling, low in storage efficiency, difficult to identify abnormities and the like. Multi-source data is obtained by means of distributed acquisition nodes, a distributed time sequence database architecture is adopted for storage, a standard data set containing a multi-dimensional index is constructed, and efficient storage and rapid retrieval of the data are achieved. Positioning target data to construct a graph model, building a layered and partitioned distributed graph database, and capturing change events in real time to realize dynamic updating. And mining a resource causal relationship by using a graph neural network, and screening an effective causal chain. Meanwhile, according to a graph model, monitoring weights of nodes and connecting edges are calculated, differential monitoring is implemented, potential abnormal points are accurately identified, and a resource scheduling strategy is generated in combination with a causal relationship. Real-time feedback adjustment and database updating are performed during execution, multi-resource deep integration and intelligent scheduling are achieved, and the resource utilization rate and stability of the system are remarkably improved.

The invention relates to the technical field of cloud computing storage, in particular to a cloud storage dynamic optimization method and system, and the method comprises the following steps: based on a cloud storage data access record, recognizing an access time interval sequence, analyzing the variation amplitude of adjacent time intervals, and calculating the standard deviation of the access time intervals; and determining the access period stability according to the standard deviation and the change amplitude, and classifying the data objects to obtain a data access period stability classification value. According to the method, task scheduling is more accurate by identifying a data access time interval sequence and changes thereof and refining data classification and storage resource allocation, a high-throughput and low-delay storage path is matched, the health condition of a storage node is evaluated, a fault node is identified in advance, data stability is guaranteed, and data copy distribution is adjusted; the storage reliability and stability are improved, the storage path for accessing the intensive data is optimized, the dynamic load balance of the storage resources is realized, and the storage efficiency is improved.

The invention discloses a network attack path prediction method and system based on a knowledge graph, and relates to the technical field of knowledge graphs, and the method comprises the steps of multi-source heterogeneous data processing, dynamic knowledge graph construction, attack path prediction, high-confidence attack path acquisition and information feedback. According to the method, space-time reference alignment of data is guaranteed, a dynamic knowledge graph is constructed to update entity attributes in real time, a double-storage mode is adopted, the storage efficiency is improved, a graph attention mechanism and an attack path prediction model are constructed, attack paths are predicted, the attack path detection and prediction precision is improved, and the attack path detection and prediction efficiency is improved. The problem of limitation in the current network attack path prediction process is solved, the threat level of the high-confidence attack path is obtained through analysis, information feedback is carried out, updating of the dynamic graph is completed, and the accuracy, reliability and authenticity of a network attack path prediction result are guaranteed.

The invention relates to the technical field of large models, in particular to a large-model-oriented reasoning service system and an inter-stage resource adjustment method, and the system comprises a pre-filling processor and a decoding processor; a computing resource controller implementing computing resource control of the prefill processor and the decode processor based on a multi-process service, the multi-process service allowing a target number of streaming multiprocessors to be designated to a prefill process representing an entity of the prefill processor and a decode process representing an entity of the decode processor; and the unified storage manager is used for managing the storage utilization rate of the system and the storage access of the pre-filling processor and the decoding processor to the large model weight and the KV cache. Therefore, the problems of delay interference, low storage efficiency and the like between pre-filling and decoding in related technologies are solved.

The invention discloses an intelligent bid evaluation expert extraction and management system, and relates to the technical field of expert extraction systems. Comprising a data cleaning and compression module, a database logic and table relation design module, an expert information collection standard and automation module, an information real-time updating and principal management module, an extraction algorithm design module, a safety and authority management module and a real-time monitoring and auditing function module. Through a self-adaptive sparse auto-encoder and a dynamic clustering technology in the data cleaning and compression module, abnormal values in a bid evaluation expert database can be detected and cleaned, data compression is performed by adopting a self-adaptive dictionary coding algorithm, and the storage efficiency and query performance of the database are improved; based on a multi-modal information fusion and reinforcement learning method, accurate collection, dynamic updating and intelligent management of expert information are realized. The randomness and fairness of the bid evaluation expert extraction process are ensured by using a mode of combining deep reinforcement learning and a quantum random number generator in an extraction algorithm design module.

The invention belongs to the technical field of computers, and provides a video display and storage method and system based on a wayland protocol and a storage medium in order to solve the problems that an existing wayland video system is high in rendering delay, high in CPU load, low in storage efficiency and poor in cross-platform compatibility. Video frames are directly written into an Overlay layer through DRM-KMS and mixed by bypassing a synthesizer, meanwhile, CPU memory amplitude overhead is eliminated through zero copy transmission, an ISP-GPU-VPU direct transmission assembly line is constructed, full-link DMA-BUF direct transmission is achieved, and delay is greatly reduced; iSP preprocessing, GPU shader scaling and VPU coding are all executed by special hardware, so that the load is reduced, and the multi-path processing capability is improved; vPU dynamic code rate compression is utilized, segmented storage is carried out according to events / time, metadata is synchronized to a database, redundant storage is avoided, and space is saved.

The invention relates to the technical field of data processing, and discloses a big data-based file analysis storage division mode and a distributed storage system.The method comprises the following steps: S1, file analysis: analyzing an input big data file, and extracting a logic structure, field content and an association relationship of the big data file; s2, data block generation: dividing an analysis result into data blocks, and generating size and relevance information of the data blocks; s3, association weight generation: establishing a weight relationship between the similarity between the data blocks and the joint access frequency; and S4, storage division optimization: based on the association weight, optimizing the distribution of the data blocks among the storage nodes. According to the method, distribution of the data blocks is optimized through the synergistic effect of the association weight generation module and the dynamic decision-making module, so that the data blocks with high association degree are preferentially distributed to the same storage node, cross-node data transmission is greatly reduced, a storage strategy can be adjusted in real time, and the storage efficiency is improved. Therefore, communication delay and network resource consumption caused by cross-node access are reduced.

The invention discloses an intelligent warehouse management method and system for hazardous wastes, and the method comprises the steps: carrying out the real-time monitoring of the state of an object in a hazardous waste warehouse through a deployment sensor network, obtaining the basic data of the position of the object and the environment parameter, and transmitting the basic data to a central processing platform, thereby obtaining a preliminary data set; performing integration processing on multi-source information by adopting a data fusion technology according to the preliminary data set, eliminating redundancy and noise, and determining a standardized data record; aiming at the standardized data record, dynamically analyzing the distribution of goods locations in the warehouse by utilizing a path optimization algorithm, and if the change of the occupation state of the goods locations is detected, adjusting a goods location allocation scheme in real time, and outputting an optimal storage location combination; according to the optimal storage position combination, a dynamic route planning scheme is generated in combination with warehouse layout information, and the optimal moving track of the operation equipment is obtained; according to the invention, intelligent management of the hazardous waste warehouse is realized, and the storage efficiency and safety are improved.

The invention relates to the technical field of data storage, in particular to a cross-border e-commerce-oriented foreign trade data storage method and system, and the method comprises the following steps: obtaining customs clearance and settlement data packets, reading time fields, detecting the consistency, judging the sequence, reordering auxiliary fields, generating a structure list, matching a currency mapping table, and constructing an archiving path. And calculating a residence time length and a repetition ratio to judge a transfer mode, performing grouping writing, establishing mapping and generating a cross-border e-commerce structured archiving storage data unit. According to the method, the consistency and the maintainability of data storage are improved by judging the synchronism and the access priority of the cross-border e-commerce platform data, optimizing integration of customs clearing data and settlement data, sorting fields and generating an asynchronous field positioning list, the storage logic is simplified by constructing a reasonable archiving path and performing segmentation management, and the storage efficiency is improved. The storage mode is flexibly adjusted according to the residence time and the repetition rate, redundant data is reduced, and the completeness and consistency of foreign trade data are ensured through accurate field binding.

The invention discloses an unsupervised semi-pairing cross-modal retrieval method and system based on deep learning, relates to the field of artificial intelligence, and is used for solving the problems of annotation data dependence, asymmetric semantic association and high-dimensional storage efficiency. According to the method, a double-branch visual encoder and a dynamic prompt text encoder are combined, dynamic weighting of visual-text features is achieved through gating cross attention, and modal redundancy interference is restrained. An enhancement strategy is generated through low-frequency semantic guidance, and the long-tail word coverage rate is increased; a dual-stage quantitative hierarchical index is constructed, coarse-grained clustering and fine-grained product quantitative compression feature storage is adopted, and million-level data real-time retrieval is supported. A degradation aware increment maintenance mechanism monitors data distribution offset through a KL divergence threshold, and triggers index reconstruction to maintain long-term update precision. According to the method, limitation of a traditional strong pairing model is broken through, cross-modal sensitive content second-level positioning is achieved, asymmetric semantic alignment is effectively solved, and retrieval efficiency is improved.

The invention relates to the technical field of database optimization, provides a massive astronomical data query processing method, and aims to solve the efficiency bottleneck of astronomical data in storage, management and query processes. By constructing a distributed spatial database, efficient storage and management of large-scale astronomical data are realized; an index structure of astronomical data is optimized by utilizing an LB-ANN index, so that query load balance and data retrieval efficiency are improved; besides, efficient fusion of structured query and unstructured query is realized through SQL-ANN mixed query in combination with a JIT compiling technology, the query execution sequence is optimized, and the query efficiency is remarkably improved; the overall method breaks through the bottlenecks of traditional astronomical data query in calculation complexity, storage efficiency and index optimization, and an accurate and efficient solution is provided for efficient intelligent processing of large-scale astronomical data.

The invention discloses a multi-scene application data acquisition method based on atomization design. The method comprises the following steps: analyzing reported data into an atomization data structure; performing semantic recognition on the data labels to generate a label semantic mapping result; constructing a layered cache structure; generating a compressed semantic mapping rule by applying a multi-level semantic difference coding algorithm; identifying an optimal query path from the tag affinity matrix; and executing historical data cleaning based on the data value. Intelligent mapping of data labels is achieved through semantic recognition and version control, the storage efficiency is improved through predictive lazy loading and compression coding, the query path is optimized based on the affinity matrix, and the technical problem of multi-scene application data collection is effectively solved.

The invention discloses an intelligent building equipment historical data storage method, and relates to the field of building equipment historical data storage, and the method comprises the following operation steps: S1, collecting and preprocessing global data; s2, intelligent storage strategy decision making; s3, designing a semantic storage structure; s4, constructing and managing a distributed storage cluster; s5, intelligently writing and indexing historical data; and S6, efficient query and deep analysis of an interface. According to the historical data storage method for the intelligent building equipment, provided by the invention, all-around improvement is realized in the aspects of data quality, storage efficiency, association capability, system reliability, operation convenience and the like through refined design of a data full link; and finally, the equipment data of the intelligent building really becomes a core asset which can be perceived, analyzed and utilized, and powerful support is provided for efficient operation, energy conservation, consumption reduction and safety guarantee of the building.

The invention provides an attachment packing and unpacking method, system and device based on a compression algorithm and a medium, and belongs to the technical field of file compression. The method comprises the steps that all attachment files needing to be packaged are obtained by traversing a specified directory, screening and filtering are conducted according to file attributes, and a renamed file list and a path mapping table are generated; according to the renamed file list, using a Brotli algorithm to compress each attachment file based on a preset compression level, and packaging the compressed data into a packaged file containing metadata, compressed data and verification information; when the packed file needs to be unpacked, the legality of the packed file is confirmed through magic number verification and index analysis, file data are recovered through Brotli streaming decompression, and the original directory structure of the file is restored. According to the method, efficient compression and streaming decompression are achieved through the Brotli algorithm, and the attachment storage efficiency, the transmission speed and the data integrity are remarkably improved in combination with the intelligent screening and directory structure reduction technology.

The invention relates to the technical field of monitoring data management, in particular to a BIM-based water conservancy project monitoring data management method. The method comprises the following steps: analyzing the stability of water conservancy data on a monitoring point time sequence, preliminarily screening out a stable reference point, further analyzing the consistency of the stable reference point and a local reference point at a moment, and screening out an available reference point; meanwhile, considering the aggregation condition of the distribution density of the screened available reference points in the region, and determining a final reference point; according to the BIM water conservancy data storage method, comprehensive evaluation on local time sequence and distribution density is carried out on the monitoring points with stable data, the necessary storage number of the monitoring points is reduced, the storage pressure is reduced, the backup number of BIM water conservancy data is effectively increased, and the BIM water conservancy data storage efficiency is improved. And the backup efficiency is further improved.

The invention discloses a mine digital asset storage management method, which comprises the following steps of: extracting characteristics of mine data, dividing the mine data into hot data, warm data and cold data according to the characteristics, selecting different storage devices for the data, and dynamically adjusting migration of the data among the different storage devices based on access frequency and business requirements; future data storage requirements are predicted through a machine learning algorithm, and storage resources are scheduled; performing full life cycle management on the data, performing business value evaluation on the data, and adjusting storage priority and resource allocation based on the data value; carrying out full-life-cycle encryption on the data, and recording data operation through a block chain technology; energy consumption of the storage equipment is monitored in real time, and storage energy efficiency and carbon footprint management are adjusted. The problems of low storage efficiency, high storage cost and insufficient energy consumption optimization caused by data type diversity, access frequency difference and insufficient storage resource utilization in mine data management can be solved.

The invention relates to the technical field of data storage, and discloses a lossless compression algorithm-based data storage method, which comprises the following steps of: receiving to-be-stored data through a data interface, automatically identifying a data format by using a data analysis tool, and performing feature analysis to obtain the size, the type and the complexity of the data; through deep analysis of data features, the method can flexibly select a proper Huffman coding processing mode, and optimizes a compression process according to different data types and complexity. In addition, a context modeling optimization mechanism is introduced, the coding mode is dynamically adjusted, and the compression ratio and the storage efficiency are further improved; efficient storage and reliability verification of data are ensured through a block storage strategy in combination with metadata management; the problem that an existing compression algorithm is low in efficiency is solved, the integrity and reliability of data storage are improved, and the method is suitable for storage requirements of large-scale data and diversified data.

The invention relates to the field of machine learning, in particular to a model weight confirmation method and device based on a block chain and a model fingerprint, and the method comprises the steps: carrying out the abstract compression of a model weight of each layer of network of a model, and obtaining a model weight abstract; performing Hash encryption and recording on the model weight abstract generated each time by adopting a Merkle tree structure to obtain a model fingerprint record; and uploading the model fingerprint record to the block chain network so as to verify the model fingerprint record. According to the invention, encryption processing is carried out on the model weight through the Hash encryption algorithm with the secret key, the model fingerprint is constructed in combination with the Merkle tree to generate a unique model fingerprint, the model fingerprint is uploaded to the block chain, the model fingerprint is combined with the block chain technology, a safe storage environment is provided for the model fingerprint, and the storage efficiency of the model fingerprint is improved. And the credibility of model ownership authentication and content tracing is enhanced.

The invention relates to the technical field of hierarchical storage, in particular to a photo-electromagnetic storage method and system based on hierarchical management, and the method comprises the following steps: collecting read-write speed, error rate, durability and cost data of optical storage and electromagnetic storage media, carrying out the sorting and grading processing of the data according to the read-write speed and the error rate, and carrying out the sorting and grading processing of the data; and performing score calculation on each storage medium to obtain a medium performance score. According to the method and the device, the data is effectively distributed to the most suitable storage medium by accurately scoring the performance of the storage medium and combining specific requirements of the to-be-stored data, so that the data access speed and efficiency are improved, the total storage cost is reduced, and particularly, when a large amount of data is processed, the data compression level and the encoding format are dynamically adjusted, and the data compression efficiency is improved. The storage efficiency of the data is further optimized, data redundancy is reduced, the overall performance of data storage and access is remarkably improved through optimization measures, and meanwhile long-term safety and integrity of the data are ensured.

The invention relates to the technical field of intelligent logistics and supply chain management, and discloses a cross-border e-commerce logistics intelligent storage scheduling and transportation cooperation method based on AI path planning, and the method comprises the steps: building a monitoring module, a dynamic information capturing module, a data classification module, a data analysis module, a storage scheduling and transportation cooperation module and a management module; the monitoring module monitors storage and transportation information in real time through a sensor and monitoring equipment, the dynamic information capture module collects external environment data and internal abnormal events, the data classification module classifies the monitored data and the captured information, and the data analysis module constructs a path planning model according to the stored data of the data classification module. The system comprises a storage scheduling and transportation collaboration module, a storage scheduling and transportation collaboration module and a management module, the storage scheduling and transportation collaboration module generates an optimal path and calculates a storage efficiency index # imgabs0 #, a transportation path lifting coefficient # imgabs1 # and a storage-transportation collaboration index # imgabs2 #, the storage scheduling and transportation collaboration module optimizes the path according to a calculation result and sets an early warning mechanism, and the management module assists management personnel in monitoring and decision making through a visual interface.

The invention discloses a data asset secure storage method based on a block chain technology, and the method comprises the following steps: data preprocessing and fragmentation storage: carrying out the preprocessing of to-be-stored original data assets, extracting metadata information, and generating a unique hash identifier; through multi-chain collaborative storage, dynamic hierarchical encryption and cross-chain redundant backup, the security of data assets is remarkably improved, fragment storage is combined with a heterogeneous block chain network, the data storage efficiency is improved, and single-point failure and centralized attack risks are avoided; the anti-cracking capability is enhanced through dynamic encryption and distributed key management; a cross-chain synchronization mechanism ensures high availability of data, so that the data can be efficiently circulated in different block chains; the operation and maintenance cost is reduced through automatic verification and recovery driven by the intelligent contract, the auditing tracking and life cycle management functions meet the compliance requirement, and the method is suitable for data storage scenes in the high-sensitivity fields of finance, medical treatment and the like.

The invention discloses a bidirectional feedback-driven large-model dynamic memory regulation and control system and method, and the system comprises a router, a generator, an actuator and a memory system, and is optimized through four core mechanisms: 1) a dynamic grading mechanism: dividing memory grades based on timeliness, accuracy, repeatability and context consistency by using fuzzy comprehensive evaluation; 2) a forgetting mechanism: performing knowledge distillation compression on the low-priority memory by adopting TinyBERT, and clearing redundant data; 3) an enhancement mechanism: combining similar task memories through LLM to generate an optimized narrative text; and 4) extracting a mechanism, and accurately retrieving knowledge in combination with a potential semantic index and retrieval enhancement generation technology, realizing closed-loop regulation and control of task allocation, generation, execution and memory enhancement through bidirectional interaction of actuator feedback and a memory system, remarkably improving storage efficiency, task solving speed and long-term self-adaptive ability, and greatly improving the system performance. The method is suitable for complex scene optimization of a large language model.

The invention relates to the technical field of urban geological survey, in particular to an urban geological survey method based on a big data cloud computing technology, which comprises the following steps: based on urban geological drilling detection data, land subsidence monitoring data and underground pipe network distribution data, grouping the data according to geographic grid units and time points, performing statistical analysis to obtain a geological parameter statistical set; and based on the geological parameter statistical set, setting multiple levels of storage partitions according to a data updating frequency, and establishing a geological data partition storage table. According to the method, urban geological data are grouped according to geographic grid units and time points, a multi-level storage partition mechanism is established, hierarchical storage and compression of the data are realized, the occupied storage space is reduced, and the data access speed is increased; key monitoring areas are screened based on a geological parameter change rate threshold, differential compression storage strategies are adopted for different areas, and the overall storage efficiency is optimized on the basis of guaranteeing important data integrity.