Patents
Literature
Patsnap Eureka AI that helps you search prior art, draft patents, and assess FTO risks, powered by patent and scientific literature data.

5742 results about "Data chunk" patented technology

Multi-modal document retrieval enhancement generation method based on large model

The invention belongs to the technical field of multi-modal data processing, and particularly relates to a multi-modal document retrieval enhancement generation method based on a large model, which comprises the following steps: receiving query content input by a user for a multi-modal document; processing the query content by adopting an embedded model, generating a query vector representing user query semantic information, and storing the query vector in a vector database; analyzing the multi-modal document to obtain long text information, segmenting the long text information into data blocks by adopting a recursive partitioning strategy, and numbering and marking the data blocks; carrying out vectorization processing on the data blocks by adopting an embedded model to generate document vectors, storing the document vectors into a vector database, and constructing a hierarchical index structure; retrieving in a vector database based on the query vector, and returning a retrieval result; and processing a retrieval result by utilizing a large language model to generate response content conforming to the query intention of the user. According to the method, the multi-modal document can be effectively analyzed and processed, and the accuracy and comprehensiveness of analysis are improved.
Owner:杭州长望智创科技有限公司

Large language model reasoning acceleration method and system based on dynamic video memory compression and memory isomerism

The invention discloses a big language model reasoning optimization method and system based on dynamic video memory compression and memory isomerism, and intelligent management of video memory resources is realized by integrating a dynamic compression strategy of KV Cache and a memory parallel architecture. The method comprises the following steps: 1) analyzing the spatial-temporal characteristics of the KV Cache in real time, adaptively selecting a quantization compression algorithm, a rarefaction algorithm or a low-rank decomposition algorithm, performing hierarchical storage based on attention head importance scores, keeping high precision of a core head, and implementing low-bit quantization on a secondary head; (2) the compressed inactive data are divided into a plurality of data blocks to be stored in a system memory, a parallel data channel group is established according to the number of physical channels, the compressed blocks are concurrently read through multiple channels during loading, and parallel decompression of a sparse matrix is accelerated through a GPU tensor core; and 3) constructing a KV Cache multiplexing mechanism and a parallel channel, and parallelizing a compression / decompression process and model calculation by adopting a hardware acceleration compression and asynchronous pipeline mechanism.
Owner:HANGZHOU AMTD YINGANG DIGITAL TECH CO LTD

Intelligent factory data processing method and system based on industrial internet

The invention provides an intelligent factory data processing method and system based on the industrial Internet, and the method comprises the steps: firstly obtaining a real-time industrial data set collected by a multi-mode sensor in a target production region, covering various data, such as equipment vibration signals, carrying out the time sequence synchronization processing of the real-time industrial data set, and generating a synchronization data block; the method comprises the following steps of: extracting dynamic state characteristics of a production line from the data, calling a pre-trained anomaly detection model to carry out multi-dimensional anomaly detection, outputting an anomaly detection result, matching a preset expert knowledge base according to the anomaly detection result, and generating a dynamic control instruction set containing equipment adjustment parameters and the like; finally, production parameter configuration is adjusted based on the dynamic control instruction set and fed back to the industrial control terminal in real time, optimization of current production resource configuration is achieved, and the production efficiency and the resource utilization rate of an intelligent factory are effectively improved.
Owner:HIMIT (SHENZHEN) TECH CO LTD

Parallel synchronization method and system for unstructured files

The invention provides an unstructured file parallel synchronization method and system, and relates to the technical field of file synchronization, and the method comprises the steps: carrying out the intelligent partitioning of a target file through a pre-training file partitioning model, and generating a double-layer cascade index number and a data hash value; and then, according to the data transmission priority model, transmitting the data sub-blocks to a target end database through a self-adaptive parallel transmission channel, and performing asynchronous verification. And finally, recombining the data sub-blocks based on the double-layer cascade index number and the data block association graph by utilizing a pre-training file recombining model, generating a synchronous target file, and submitting confirmation information to the block chain network. According to the method, the unstructured file synchronization efficiency can be improved, the data integrity and security are guaranteed, and the network transmission pressure and the storage cost are reduced.
Owner:北京科杰科技有限公司

Time sequence database data compression method, system and device and storage medium

The invention relates to the technical field of wind power generation equipment data processing. The invention provides a time sequence database data compression method, system and device and a storage medium. The method comprises the steps of dynamically partitioning time sequence data based on a preset time window; hybrid coding compression is performed on each data block, and a composite strategy of difference coding and run length coding is adopted for numerical data; a metadata index of the compression blocks is established, the time range, the data feature statistics and the compression parameters of each data block are recorded, and the metadata index comprises extreme value distribution, variance features and data fluctuation frequency indexes; and dynamically adjusting a compression strategy according to a historical data feature analysis result, predicting data fluctuation modes of different equipment sensors through a machine learning model, and automatically selecting an optimal coding combination and compression granularity for subsequent data blocks. The problems that when wind power plant time sequence data are processed through an existing method, time sequence characteristics are difficult to adapt, the storage cost is high and the query efficiency is low are solved.
Owner:HUANENG DINGBIAN NEW ENERGY POWER GENERATION CO LTD +1

Operator optimization method, electronic device, storage medium and program product

The invention relates to the technical field of artificial intelligence chips, and provides an operator optimization method, electronic equipment, a storage medium and a program product, and the method comprises the steps: determining the block size of operator data in each dimension based on the batch size and mask mode of the operator data and the hardware parameters of computing equipment, each dimension comprises a batch dimension and a sequence length dimension; segmenting the operator data based on the block size of the operator data in each dimension to obtain a plurality of data blocks; and distributing the calculation tasks corresponding to the plurality of data blocks to a plurality of processing units on the calculation equipment, and performing parallel execution on the calculation tasks corresponding to the plurality of data blocks based on the plurality of processing units. According to the method and the device, the data are segmented in the batch dimension and the sequence length dimension at the same time, so that when the data batch is small, a plurality of processing units on the computing equipment can also participate in computing at the same time, hardware resources are prevented from being idle, the utilization rate of the hardware resources is improved, and the overall computing efficiency is improved.
Owner:SHANGHAI BIREN TECH CO LTD

Distributed storage and indexing method

The invention relates to the technical field of information retrieval, and discloses a distributed storage and indexing method, which comprises the following steps of: dynamically fragmenting an original file through an improved consistent Hash algorithm to generate a plurality of data blocks with timestamps; constructing a three-dimensional Bloom filter index matrix containing timestamps, data types and content features for the data blocks with the timestamps, associating a B + tree local index with an inverted global index through a hierarchical index structure, and establishing an index update priority queue by adopting a two-channel synchronization mechanism, performing real-time increment synchronization on the hotspot index through a heartbeat mechanism, performing batch synchronization on the cold data layer index according to a cold data synchronization period, predicting a data distribution probability through a distributed query statistics probability table during query, and initiating multi-path query in parallel based on probability weight, and the query path is dynamically optimized according to the node group storage medium type and the inter-node network transmission delay, so that rapid distribution adjustment and access of distributed storage are realized.
Owner:SHENYANG LIUFANG INFORMATION TECHNOLOGY CO LTD

Data full-flash storage optimization method and system based on cloud computing

The invention provides a data all-flash storage optimization method and system based on cloud computing, and the method comprises the steps: receiving a storage operation request stream of a user terminal through collecting a multi-dimensional performance data set of each node of an all-flash storage cluster in real time, wherein the multi-dimensional performance data set comprises a storage space fragmentation rate, an input / output request queue depth and a network link bandwidth utilization rate; analyzing a to-be-stored data block identifier set and an operation mode label, traversing a historical access record library based on to-be-stored data block identifiers, extracting historical access time sequence characteristics and physical storage position proximity of associated data blocks, and combining the multi-dimensional performance data and the historical access time sequence characteristics to obtain a data block identifier set; according to the method, a dynamic access popularity prediction model in a preset time period is constructed, and finally, a cross-node fragmentation storage topological graph and a cache preloading strategy matrix are generated according to the proximity of the dynamic access popularity prediction model and a physical storage position, so that the optimization of all-flash storage is realized, and the data storage performance in a cloud computing environment is improved.
Owner:YUGE ELECTRONIC TECH CO LTD

Heterogeneous computing thread block optimal scheduling method and system based on dynamic topology mapping

The invention belongs to the field of parallel computing architecture optimization, and relates to a matrix multiplication acceleration method and system based on dynamic computing resource mapping, and the method comprises the steps: constructing a dynamic topology model driven by tensor dimension features, and generating a thread block distribution mode according to matrix parameters and GPU hardware information; constructing a multi-dimensional resource scheduling strategy library, dynamically selecting an optimal thread block distribution strategy from the multi-dimensional resource scheduling strategy library, and generating a binding relationship between the thread blocks and the data blocks; calculating collaborative access logic of thread blocks and storage hierarchies based on block parameters and dynamic mapping function optimization; distributed calculation is carried out, calculation and data transmission are parallelized through pipelining and a double-buffering mechanism, and result aggregation across calculation units is completed synchronously through atomic operation and a barrier. According to the method, discontinuous memory access conflicts can be effectively reduced, the execution efficiency of the calculation instruction and the utilization rate of the cache space are improved, the parallel calculation process of accelerating and optimizing the general matrix multiplication is realized, and the data processing efficiency is improved.
Owner:SOUTH CHINA UNIV OF TECH

RAID card static cache management method and device based on data popularity

The invention relates to the technical field of data storage and processing, in particular to an RAID card static cache management method and device based on data popularity, and the method comprises the steps that the physical position of a data block needing to be accessed in an SSD array is acquired according to a read-write request; historical data access information of the RAID card is collected, a historical data set is generated, and a neural network model is trained by using the historical data set to obtain a cache management model; predicting and outputting data blocks which are possibly accessed in the future and the access probability of the data blocks through the cache management model; the RAID controller dynamically adjusts a cache strategy in combination with an LRU strategy according to a prediction result output by the cache management model and a read-write request of a file system, and optimizes a storage position and an updating mechanism of a data block in a cache space; and when the hit rate or the data access efficiency does not reach the preset value, adjusting the parameters of the cache strategy and updating the cache strategy. According to the method, cache resources can be more effectively distributed, the cache hit rate is improved, the cache replacement overhead is reduced, and therefore the performance of a storage system is optimized.
Owner:SOUTH CHINA UNIV OF TECH

File uploading method and device, storage medium, electronic equipment and program product

The invention discloses a file uploading method and device, a storage medium, electronic equipment and a program product, and relates to the technical field of computer networks and distributed storage, and the method comprises the steps: collecting metadata of a to-be-uploaded file and a network state of an uploading path between a client and a server in real time; calculating the size of a target fragment of the to-be-uploaded file for data uploading between the client and the server through the metadata and the network state; segmenting the to-be-uploaded file based on the target fragment size to obtain a plurality of data blocks; according to the method, the plurality of data blocks are used for uploading the file from the client to the server, and the uploading parameters of the different data blocks are recorded, so that the uploading result of the file to be uploaded transmitted to the server is determined through the uploading parameters, the technical problem that the uploading efficiency and robustness of the object storage system are poor is solved, and the uploading efficiency of the object storage system is improved. The technical effect of improving the uploading efficiency and robustness of the object storage system is achieved.
Owner:JINAN INSPUR DATA TECH CO LTD

Dynamic data distribution storage method and system for RAID (Redundant Array of Independent Disks)

The invention relates to the technical field of data storage, in particular to a dynamic data distribution storage method and system for an RAID (redundant array of independent disks), and the method comprises the following steps: based on the access frequency of data blocks in a disk, recording the access frequency and timestamp of each data block, carrying out heat rating on the data blocks, and determining thresholds of hot data and cold data. According to the method, the dynamic distribution of data storage is effectively optimized by analyzing the access frequency of the data blocks, and the performance and the economic efficiency of a storage system are remarkably improved by migrating the frequently accessed data to the cache disk and migrating the less accessed data to the low-cost storage; according to the method, unnecessary occupation of high-performance storage resources is reduced, the operation cost is also reduced, heat rating is performed on the data blocks, and the storage positions of the data blocks are correspondingly adjusted, so that the data processing speed is higher, meanwhile, the data security is improved, when a hard disk fails, data recovery is faster, and the service continuity is ensured.
Owner:SHANGHAI FAITH INFORMATION TECH CO LTD

Data association maintenance processing method and system based on file reference

The invention provides a data association maintenance processing method and system based on file reference, and relates to the technical field of data association maintenance, which comprises the following steps: constructing a directed reference graph, identifying direct and indirect reference paths, and constructing a reference dependence intensity matrix in combination with a time sequence, an interaction amount and a reference propagation probability. And carrying out file clustering based on the matrix, and determining a core reference file set. And when the core file is updated, acquiring incremental update data by adopting data block level difference comparison, and merging the data blocks based on the content similarity and the semantic association degree. After the merged data blocks are segmented according to threshold values and semantic boundaries, updating is propagated to a target node through a multi-path updating propagation network, and the data consistency is ensured through version consistency check and a compensation updating mechanism. According to the method, data association between files can be effectively maintained, and the data updating efficiency and consistency are improved.
Owner:NORTH CHINA MUNICIPAL ENG DESIGN & RES INST

Large model data distributed management method and device, equipment and storage medium

The invention discloses a large model data distributed management method and device, equipment and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: segmenting large model training data to obtain a plurality of data blocks, and determining a predicted access frequency based on a long short-term memory network model and the historical access frequency of the data blocks; caching the large model training data corresponding to the data block to a corresponding data cache layer by utilizing the predicted access frequency; constructing a resource portrait by using the static attribute and the dynamic index of the GPU node, and allocating the large model training task to a target GPU node by using a preset hybrid strategy and the resource portrait based on the predicted access frequency and the storage position corresponding to the data block in the data cache layer; when it is monitored that the large model training task on the target GPU node is executed, periodic snapshot is conducted on the large model training task through a distributed snapshot algorithm, and the obtained complete data state is stored in a distributed storage center. And the resource vacancy rate is reduced.
Owner:SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD

Multi-scene adaptive communication method and system of VOIP product

The invention relates to the technical field of VOIP product communication, and discloses a multi-scene adaptive communication method and system of a VOIP product. The method comprises the following steps: carrying out collection and feature extraction on network state parameters to obtain a scene category, and calling reference communication parameters corresponding to the scene category from a scene parameter library; calculating a composite cumulative distribution function based on the reference communication parameters to obtain an optimized communication parameter combination; applying the optimized communication parameter combination to an edge cloud collaborative coding and decoding framework, and carrying out coding and decoding processing on audio and video data to obtain coded communication data; performing block processing on the encoded communication data, and determining a storage position and a communication node of a data block; according to the method, the communication nodes are divided into the master node and the slave node, and the master node and the slave node are subjected to multi-objective optimization to obtain the data transmission path, so that the dynamic optimization of the communication parameters is realized, a VOIP product has scene self-adaptive capability, and the communication service quality of the VOIP product is improved.
Owner:SHENZHEN SKYRISE TECH CO LTD

Agile remote sensing satellite on-satellite full-autonomous task planning system

The invention discloses an on-satellite full-autonomous task planning system for an agile remote sensing satellite. Comprising an imaging target database module, a high-precision orbit prediction and orbit calculation module, a target visible arc section calculation module, an imaging task preprocessing module, a subsystem constraint model module, an imaging task planning module, a data transmission task planning module, a solid memory autonomous erasing module, an instruction data block generation module and a satellite state real-time monitoring module. By carrying out constraint modeling on subsystems or single machines such as load, data transmission, control and the like, an on-satellite full-autonomous operation process from receiving of subsystem or single machine instruction blocks from user requirements is established, satellite control is effectively simplified, and remote control uploading efficiency and on-satellite autonomous operation capacity are improved. And through a satellite state real-time monitoring module, the execution condition of tasks on the satellite, satellite state changes and the like are monitored in real time, task execution stopping, whole satellite sun-facing or re-planning are further triggered, task requirements and environment changes are quickly responded, task plans are formulated and adjusted in real time, and the task execution efficiency is improved.
Owner:BEIJING INST OF CONTROL ENG

Solid state disk data reading method and system, storage medium and program product

The invention discloses a solid state disk data reading method and system, a storage medium and a program product, and the method comprises the steps: calculating a temperature change rate and a temperature accumulation effect coefficient of a to-be-read data block according to a current temperature value and a historical temperature record; dividing the physical page address into a plurality of subintervals, and determining a loss level; calculating an initial reading voltage threshold value; compensating the initial reading voltage threshold value to obtain a target reading voltage threshold value; performing parallel reading on the to-be-read data block by adopting the target reading voltage threshold value to obtain original reading data; performing crosstalk correction on the original read data to obtain corrected data; when the reliability verification result of the corrected data of any subinterval is lower than a preset threshold value, error correction processing is carried out; and updating the loss level of the sub-interval according to the verification result of the reliability verification, and re-dividing the sub-interval of the data block to be read. The data reading delay is reduced, and the performance of the solid state disk in a scene of frequently reading small files is improved.
Owner:SHENZHEN WEIKEWEIYE ELECTRONIC TECH CO LTD

Database load optimization method and device based on dynamic fragmentation, equipment and medium

The invention relates to the technical field of big data and the field of financial science and technology, and discloses a database load optimization method based on dynamic fragmentation, which comprises the following steps: establishing an association relationship between fragments and storage nodes by configuring basic logic fragments and dynamic logic fragments; monitoring a data access record, identifying hotspot data and marking the hotspot data as a to-be-migrated data block; when the fragment load exceeds a threshold value, migrating the high-load data block to the dynamic logic fragment; and when the access frequency is reduced, moving the low-load data block back to the basic logic fragment. According to the method, through configuration and management of the dynamic logic fragments, in combination with real-time monitored access records and load state analysis, the high-load data blocks are recognized and migrated to the dynamic logic fragments for storage, and the overload problem of the basic logic fragments is relieved. Meanwhile, the low-load data blocks are moved back to the basic logic fragments, dynamic adjustment and balance of storage resources are further achieved, the load balancing capacity of the system is improved, and the requirements for high concurrency and real-time load changes in a financial scene are met.
Owner:JINZHUAN INFORMATION TECHNOLOGY CO LTD

Operator parallel processing method, electronic equipment, storage medium and program product

The invention relates to the technical field of artificial intelligence, and provides an operator parallel processing method, electronic equipment, a storage medium and a program product, and the method comprises the steps: determining a step of data dependence based on each processing step of a first operator and a second operator; and performing parallel execution on each processing step of the first operator and the second operator by applying a synchronization strategy and a segmentation strategy based on the plurality of cooperative thread bundles. According to the method, the parallel fusion processing of operators is realized through the steps of determining data dependence, segmenting the data, realizing parallel execution by utilizing the cooperative thread bundle, inserting synchronization between the data dependence steps and the like, the calculation performance is remarkably improved, and meanwhile, the data blocks after segmentation are multiplexed, so that the data processing efficiency is improved. The data dependence in the operators and between the operators can be reduced, and the overhead of data reading, instruction synchronization and the like can be reduced.
Owner:SHANGHAI BIREN TECH CO LTD

Electronic data storage verification method and system based on hierarchical hash and smart contract

The invention provides an electronic data evidence storage verification method and system based on hierarchical hash and an intelligent contract, and relates to the technical field of block chains and data security, and the method comprises the steps: obtaining to-be-stored electronic data; to-be-stored electronic data is divided into data blocks according to a preset rule, a unique identifier and a timestamp are added to each data block, and a standardized data unit is generated; performing hierarchical hash calculation on the standardized data unit, generating a leaf node hash value by each data block, aggregating the leaf node hash values layer by layer according to a Merkle tree structure, and generating root hash; calling a block chain smart contract, submitting the tree structure hash, the timestamp and the public key encrypted signature to a block chain network, after verification of a consensus node, writing the evidence information into the block chain, analyzing a hash path submitted by a user by the smart contract to perform data verification, if the hash path is consistent with the data verification, returning verification success, otherwise, triggering an exception handling contract; through the dynamic partitioning algorithm and the layered hash structure, the calculation overhead of large file processing is reduced.
Owner:SHANDONG UNIV OF POLITICAL SCI & LAW

Distributed account book remote data synchronization method and system

The invention relates to the technical field of data synchronization, and discloses a distributed account book remote data synchronization method and system, and the method comprises the steps: carrying out the Hash mapping calculation of all nodes of a distributed account book system, and obtaining a distributed storage architecture; performing hierarchical network construction on the nodes, and setting an inter-region remote synchronization link and an intra-region local synchronization link to obtain a double-layer communication architecture; performing parallel compression coding, digital signature generation, transmission scheduling arrangement and consistency verification processing on the data blocks to obtain a node synchronous data set; calculating a correlation coefficient of node processing delay and network transmission delay to obtain a synchronous delay analysis model; based on a data block corresponding to the synchronous delay analysis model, adding a time sequence identifier and calculating an integrity check code, and generating a synchronous reliability guarantee mechanism; according to the method, the consistency and the traceability of the distributed account book data are ensured.
Owner:SHENZHEN LINGZHU TECH CO LTD +1

File high-speed transmission system based on multilink aggregation

PendingCN120358230ATransmissionHigh level techniquesFile transmissionDynamic load balancing algorithm
The invention provides a file high-speed transmission method based on multi-link aggregation, and solves the problems of low transmission efficiency and insufficient stability of a traditional single link. The method comprises the following steps: detecting network link performance parameters, and screening available links; according to the number of links, the average bandwidth and # imgabs0 # limitation, dynamically segmenting a large file into adaptive data blocks, numbering the data blocks, and adding information such as a position to a head; an improved dynamic load balancing algorithm is adopted, link quality (including bandwidth utilization rate, delay, packet loss rate, stability and dynamic adjustment of weight along with a network environment) and data block priority (distinguished according to types and sizes and the weight is adjusted according to file characteristics) are synthesized, data blocks are distributed to different links for parallel transmission, and pre-distribution verification ensures link load safety; and recombining the data blocks by the receiving end according to the numbers to realize file recovery. The system significantly improves the transmission efficiency through multi-link aggregation and dynamic strategy adjustment.
Owner:南京通达海软件有限公司

Page progressive rendering method and system based on streaming data

The invention relates to the technical field of page rendering, and discloses a progressive page rendering method and system based on streaming data, and the method comprises the steps: carrying out the data segmentation through obtaining a data stream, user interaction data and equipment performance parameters, and obtaining data blocks; then, performing cache management in combination with the data, and constructing a multi-level cache pool; thirdly, performing priority grading and sorting on the data blocks to form a rendering task queue; and according to the equipment performance parameters, optimizing a task sequence and obtaining an optimized task sequence. Next, combining the optimized task sequence and user interaction data, predicting data about to enter a viewport, and generating a viewport pre-rendering task; and finally, according to the viewport pre-rendering task, the multi-level cache pool and the equipment performance parameters, performing rendering strategy optimization to obtain a dynamically adjusted rendering task flow. The method can realize dynamic resource scheduling.
Owner:DEEP BLUE INTERNET (BEIJING) TECHNOLOGY CO LTD

Object-based storage with garbage collection and data consolidation

Embodiments are directed to a file system that include object stores. An object store for write requests may be provided. Write ahead log (WAL) entries that include data blocks may be generated. A WAL object may be generated based on the WAL entries and stored in the object store. An in-memory overlay may be updated to associate the data blocks with the WAL object. A checkpoint operation may be executed to: generate an index object that includes index entries that associate other data blocks with data objects stored in the object store; update the index object to include index entries that associate the data blocks with the WAL object; store the updated index object in the object store; update the in-memory overlay to remove the association of the data blocks and the WAL object and update the in memory WAL to remove records of successfully checkpointed WAL objects.
Owner:QUMULO INC

Data security method and system based on privacy calculation and multifunctional encryption

The invention discloses a data security method and system based on privacy computing and multifunctional encryption, and relates to the technical field of privacy security computing. Verifying the operation authority of the computing node on the encrypted data block through a zero-knowledge proof protocol, and generating a strategy compliance proof; a differential privacy budget real-time verification mechanism is adopted to verify the privacy protection intensity of strategy compliance proof, and a temporary permission token which passes authorization is obtained; based on the temporary permission token, performing encryption calculation allowed by strategy compliance proof on the encrypted data block through homomorphic operation, and generating an encryption calculation result and a calculation correctness proof; carrying out joint cryptographic packaging on an encryption calculation result, a strategy compliance proof and a calculation correctness proof, and outputting security data through a verifiable calculation protocol; according to the invention, through a differential privacy budget real-time verification mechanism and homomorphic and cryptomorphic calculation based on the temporary permission token, dual guarantee of privacy protection and data security is realized.
Owner:JIANGXI SHUDUN INFORMATION TECH NETWORK SECURITY RES INST CO LTD

Big data-based file analysis, storage and division mode and distributed storage system

The invention relates to the technical field of data processing, and discloses a big data-based file analysis storage division mode and a distributed storage system.The method comprises the following steps: S1, file analysis: analyzing an input big data file, and extracting a logic structure, field content and an association relationship of the big data file; s2, data block generation: dividing an analysis result into data blocks, and generating size and relevance information of the data blocks; s3, association weight generation: establishing a weight relationship between the similarity between the data blocks and the joint access frequency; and S4, storage division optimization: based on the association weight, optimizing the distribution of the data blocks among the storage nodes. According to the method, distribution of the data blocks is optimized through the synergistic effect of the association weight generation module and the dynamic decision-making module, so that the data blocks with high association degree are preferentially distributed to the same storage node, cross-node data transmission is greatly reduced, a storage strategy can be adjusted in real time, and the storage efficiency is improved. Therefore, communication delay and network resource consumption caused by cross-node access are reduced.
Owner:BEIJING HANXINSHENG TECH CO LTD

Solid state disk bad block wear balancing method and system, medium and program product

The invention discloses a solid state disk bad block wear balancing method and system, a medium and a program product, and relates to the field of electrical digital data process.The method comprises the steps that a storage area of a solid state disk is divided into a plurality of sub-areas, and a sub-area distribution table is generated; establishing a data block rotation link in each sub-region based on the sub-region distribution table and recording the data block rotation link as a rotation sequence table; executing a data block rotation operation according to a preset period, and recording a wear state of each sub-region; when a newly added bad block is detected, marking the sub-region containing the bad block as a to-be-migrated source region in the sub-region distribution table; determining a target region from other sub-regions according to the number of bad blocks and wear states recorded in the sub-region distribution table, and establishing a mapping relationship between the to-be-migrated source region and the target region; and migrating the data in the to-be-migrated sub-region to the target region according to the sequence in the rotation sequence table based on the mapping relationship and the rotation sequence table. The utilization efficiency of the standby block is improved, and the service life of the solid state disk is further prolonged.
Owner:SHENZHEN QINGFEN TINGXIU INFORMATION TECHNOLOGY CO LTD

Information extraction system for unstructured documents using retrieval augmentation providing source traceability and error control

A system for extracting a number of data elements from one or more unstructured data sources. The system may separate the text from the tables in a document, such that only the table data may be sent to the large language model (LLM), when the LLM only needs to review the table data. The system generates chunks from the document. The system associates unique identifiers with each chunk to provide traceability. The system identifies relevant chunks from the documents and includes the relevant chunks with a request to extract the data elements in a prompt to the LLM. The system also includes a request for the LLM to report the chunks used during extraction of the data elements. The reported chunks are stored with the extracted data for verification, auditing, and error control.
Owner:AMERICAN INTERNATIONAL GROUP INC

Data dynamic migration storage method and device, equipment and medium

The invention discloses a data dynamic migration and storage method, device and equipment and a medium, and relates to the technical field of computers. The method comprises the steps that environment parameters of a current mechanical hard disk in a distributed cluster are collected; wherein the environment parameters comprise a vibration parameter, a temperature parameter and a noise parameter; quantifying the input and output delay performance of the current mechanical hard disk by using the environmental parameters of the current mechanical hard disk; if the input and output delay performance of the current mechanical hard disk is greater than a preset threshold value, searching a target mechanical hard disk combination in the distributed cluster; and migrating and storing the target data block of the current mechanical hard disk into the target mechanical hard disk combination based on an erasure code and a multi-copy mechanism. Through the scheme, the reliability of storing the data in the mechanical hard disk in the distributed cluster is remarkably improved.
Owner:JINAN INSPUR DATA TECH CO LTD