4919 results about "Data chunk" patented technology

The invention discloses a big language model reasoning optimization method and system based on dynamic video memory compression and memory isomerism, and intelligent management of video memory resources is realized by integrating a dynamic compression strategy of KV Cache and a memory parallel architecture. The method comprises the following steps: 1) analyzing the spatial-temporal characteristics of the KV Cache in real time, adaptively selecting a quantization compression algorithm, a rarefaction algorithm or a low-rank decomposition algorithm, performing hierarchical storage based on attention head importance scores, keeping high precision of a core head, and implementing low-bit quantization on a secondary head; (2) the compressed inactive data are divided into a plurality of data blocks to be stored in a system memory, a parallel data channel group is established according to the number of physical channels, the compressed blocks are concurrently read through multiple channels during loading, and parallel decompression of a sparse matrix is accelerated through a GPU tensor core; and 3) constructing a KV Cache multiplexing mechanism and a parallel channel, and parallelizing a compression / decompression process and model calculation by adopting a hardware acceleration compression and asynchronous pipeline mechanism.

The invention provides an intelligent factory data processing method and system based on the industrial Internet, and the method comprises the steps: firstly obtaining a real-time industrial data set collected by a multi-mode sensor in a target production region, covering various data, such as equipment vibration signals, carrying out the time sequence synchronization processing of the real-time industrial data set, and generating a synchronization data block; the method comprises the following steps of: extracting dynamic state characteristics of a production line from the data, calling a pre-trained anomaly detection model to carry out multi-dimensional anomaly detection, outputting an anomaly detection result, matching a preset expert knowledge base according to the anomaly detection result, and generating a dynamic control instruction set containing equipment adjustment parameters and the like; finally, production parameter configuration is adjusted based on the dynamic control instruction set and fed back to the industrial control terminal in real time, optimization of current production resource configuration is achieved, and the production efficiency and the resource utilization rate of an intelligent factory are effectively improved.

The invention relates to the technical field of wind power generation equipment data processing. The invention provides a time sequence database data compression method, system and device and a storage medium. The method comprises the steps of dynamically partitioning time sequence data based on a preset time window; hybrid coding compression is performed on each data block, and a composite strategy of difference coding and run length coding is adopted for numerical data; a metadata index of the compression blocks is established, the time range, the data feature statistics and the compression parameters of each data block are recorded, and the metadata index comprises extreme value distribution, variance features and data fluctuation frequency indexes; and dynamically adjusting a compression strategy according to a historical data feature analysis result, predicting data fluctuation modes of different equipment sensors through a machine learning model, and automatically selecting an optimal coding combination and compression granularity for subsequent data blocks. The problems that when wind power plant time sequence data are processed through an existing method, time sequence characteristics are difficult to adapt, the storage cost is high and the query efficiency is low are solved.

The invention relates to the technical field of artificial intelligence chips, and provides an operator optimization method, electronic equipment, a storage medium and a program product, and the method comprises the steps: determining the block size of operator data in each dimension based on the batch size and mask mode of the operator data and the hardware parameters of computing equipment, each dimension comprises a batch dimension and a sequence length dimension; segmenting the operator data based on the block size of the operator data in each dimension to obtain a plurality of data blocks; and distributing the calculation tasks corresponding to the plurality of data blocks to a plurality of processing units on the calculation equipment, and performing parallel execution on the calculation tasks corresponding to the plurality of data blocks based on the plurality of processing units. According to the method and the device, the data are segmented in the batch dimension and the sequence length dimension at the same time, so that when the data batch is small, a plurality of processing units on the computing equipment can also participate in computing at the same time, hardware resources are prevented from being idle, the utilization rate of the hardware resources is improved, and the overall computing efficiency is improved.

The invention relates to the technical field of information retrieval, and discloses a distributed storage and indexing method, which comprises the following steps of: dynamically fragmenting an original file through an improved consistent Hash algorithm to generate a plurality of data blocks with timestamps; constructing a three-dimensional Bloom filter index matrix containing timestamps, data types and content features for the data blocks with the timestamps, associating a B + tree local index with an inverted global index through a hierarchical index structure, and establishing an index update priority queue by adopting a two-channel synchronization mechanism, performing real-time increment synchronization on the hotspot index through a heartbeat mechanism, performing batch synchronization on the cold data layer index according to a cold data synchronization period, predicting a data distribution probability through a distributed query statistics probability table during query, and initiating multi-path query in parallel based on probability weight, and the query path is dynamically optimized according to the node group storage medium type and the inter-node network transmission delay, so that rapid distribution adjustment and access of distributed storage are realized.

The invention provides a data all-flash storage optimization method and system based on cloud computing, and the method comprises the steps: receiving a storage operation request stream of a user terminal through collecting a multi-dimensional performance data set of each node of an all-flash storage cluster in real time, wherein the multi-dimensional performance data set comprises a storage space fragmentation rate, an input / output request queue depth and a network link bandwidth utilization rate; analyzing a to-be-stored data block identifier set and an operation mode label, traversing a historical access record library based on to-be-stored data block identifiers, extracting historical access time sequence characteristics and physical storage position proximity of associated data blocks, and combining the multi-dimensional performance data and the historical access time sequence characteristics to obtain a data block identifier set; according to the method, a dynamic access popularity prediction model in a preset time period is constructed, and finally, a cross-node fragmentation storage topological graph and a cache preloading strategy matrix are generated according to the proximity of the dynamic access popularity prediction model and a physical storage position, so that the optimization of all-flash storage is realized, and the data storage performance in a cloud computing environment is improved.

The invention belongs to the field of parallel computing architecture optimization, and relates to a matrix multiplication acceleration method and system based on dynamic computing resource mapping, and the method comprises the steps: constructing a dynamic topology model driven by tensor dimension features, and generating a thread block distribution mode according to matrix parameters and GPU hardware information; constructing a multi-dimensional resource scheduling strategy library, dynamically selecting an optimal thread block distribution strategy from the multi-dimensional resource scheduling strategy library, and generating a binding relationship between the thread blocks and the data blocks; calculating collaborative access logic of thread blocks and storage hierarchies based on block parameters and dynamic mapping function optimization; distributed calculation is carried out, calculation and data transmission are parallelized through pipelining and a double-buffering mechanism, and result aggregation across calculation units is completed synchronously through atomic operation and a barrier. According to the method, discontinuous memory access conflicts can be effectively reduced, the execution efficiency of the calculation instruction and the utilization rate of the cache space are improved, the parallel calculation process of accelerating and optimizing the general matrix multiplication is realized, and the data processing efficiency is improved.

The invention relates to the technical field of data storage and processing, in particular to an RAID card static cache management method and device based on data popularity, and the method comprises the steps that the physical position of a data block needing to be accessed in an SSD array is acquired according to a read-write request; historical data access information of the RAID card is collected, a historical data set is generated, and a neural network model is trained by using the historical data set to obtain a cache management model; predicting and outputting data blocks which are possibly accessed in the future and the access probability of the data blocks through the cache management model; the RAID controller dynamically adjusts a cache strategy in combination with an LRU strategy according to a prediction result output by the cache management model and a read-write request of a file system, and optimizes a storage position and an updating mechanism of a data block in a cache space; and when the hit rate or the data access efficiency does not reach the preset value, adjusting the parameters of the cache strategy and updating the cache strategy. According to the method, cache resources can be more effectively distributed, the cache hit rate is improved, the cache replacement overhead is reduced, and therefore the performance of a storage system is optimized.

The invention discloses a file uploading method and device, a storage medium, electronic equipment and a program product, and relates to the technical field of computer networks and distributed storage, and the method comprises the steps: collecting metadata of a to-be-uploaded file and a network state of an uploading path between a client and a server in real time; calculating the size of a target fragment of the to-be-uploaded file for data uploading between the client and the server through the metadata and the network state; segmenting the to-be-uploaded file based on the target fragment size to obtain a plurality of data blocks; according to the method, the plurality of data blocks are used for uploading the file from the client to the server, and the uploading parameters of the different data blocks are recorded, so that the uploading result of the file to be uploaded transmitted to the server is determined through the uploading parameters, the technical problem that the uploading efficiency and robustness of the object storage system are poor is solved, and the uploading efficiency of the object storage system is improved. The technical effect of improving the uploading efficiency and robustness of the object storage system is achieved.

The invention provides a data association maintenance processing method and system based on file reference, and relates to the technical field of data association maintenance, which comprises the following steps: constructing a directed reference graph, identifying direct and indirect reference paths, and constructing a reference dependence intensity matrix in combination with a time sequence, an interaction amount and a reference propagation probability. And carrying out file clustering based on the matrix, and determining a core reference file set. And when the core file is updated, acquiring incremental update data by adopting data block level difference comparison, and merging the data blocks based on the content similarity and the semantic association degree. After the merged data blocks are segmented according to threshold values and semantic boundaries, updating is propagated to a target node through a multi-path updating propagation network, and the data consistency is ensured through version consistency check and a compensation updating mechanism. According to the method, data association between files can be effectively maintained, and the data updating efficiency and consistency are improved.

The invention discloses a large model data distributed management method and device, equipment and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: segmenting large model training data to obtain a plurality of data blocks, and determining a predicted access frequency based on a long short-term memory network model and the historical access frequency of the data blocks; caching the large model training data corresponding to the data block to a corresponding data cache layer by utilizing the predicted access frequency; constructing a resource portrait by using the static attribute and the dynamic index of the GPU node, and allocating the large model training task to a target GPU node by using a preset hybrid strategy and the resource portrait based on the predicted access frequency and the storage position corresponding to the data block in the data cache layer; when it is monitored that the large model training task on the target GPU node is executed, periodic snapshot is conducted on the large model training task through a distributed snapshot algorithm, and the obtained complete data state is stored in a distributed storage center. And the resource vacancy rate is reduced.

The invention provides an electronic data evidence storage verification method and system based on hierarchical hash and an intelligent contract, and relates to the technical field of block chains and data security, and the method comprises the steps: obtaining to-be-stored electronic data; to-be-stored electronic data is divided into data blocks according to a preset rule, a unique identifier and a timestamp are added to each data block, and a standardized data unit is generated; performing hierarchical hash calculation on the standardized data unit, generating a leaf node hash value by each data block, aggregating the leaf node hash values layer by layer according to a Merkle tree structure, and generating root hash; calling a block chain smart contract, submitting the tree structure hash, the timestamp and the public key encrypted signature to a block chain network, after verification of a consensus node, writing the evidence information into the block chain, analyzing a hash path submitted by a user by the smart contract to perform data verification, if the hash path is consistent with the data verification, returning verification success, otherwise, triggering an exception handling contract; through the dynamic partitioning algorithm and the layered hash structure, the calculation overhead of large file processing is reduced.

The invention provides a file high-speed transmission method based on multi-link aggregation, and solves the problems of low transmission efficiency and insufficient stability of a traditional single link. The method comprises the following steps: detecting network link performance parameters, and screening available links; according to the number of links, the average bandwidth and # imgabs0 # limitation, dynamically segmenting a large file into adaptive data blocks, numbering the data blocks, and adding information such as a position to a head; an improved dynamic load balancing algorithm is adopted, link quality (including bandwidth utilization rate, delay, packet loss rate, stability and dynamic adjustment of weight along with a network environment) and data block priority (distinguished according to types and sizes and the weight is adjusted according to file characteristics) are synthesized, data blocks are distributed to different links for parallel transmission, and pre-distribution verification ensures link load safety; and recombining the data blocks by the receiving end according to the numbers to realize file recovery. The system significantly improves the transmission efficiency through multi-link aggregation and dynamic strategy adjustment.

The invention relates to the technical field of page rendering, and discloses a progressive page rendering method and system based on streaming data, and the method comprises the steps: carrying out the data segmentation through obtaining a data stream, user interaction data and equipment performance parameters, and obtaining data blocks; then, performing cache management in combination with the data, and constructing a multi-level cache pool; thirdly, performing priority grading and sorting on the data blocks to form a rendering task queue; and according to the equipment performance parameters, optimizing a task sequence and obtaining an optimized task sequence. Next, combining the optimized task sequence and user interaction data, predicting data about to enter a viewport, and generating a viewport pre-rendering task; and finally, according to the viewport pre-rendering task, the multi-level cache pool and the equipment performance parameters, performing rendering strategy optimization to obtain a dynamically adjusted rendering task flow. The method can realize dynamic resource scheduling.

The invention discloses a data security method and system based on privacy computing and multifunctional encryption, and relates to the technical field of privacy security computing. Verifying the operation authority of the computing node on the encrypted data block through a zero-knowledge proof protocol, and generating a strategy compliance proof; a differential privacy budget real-time verification mechanism is adopted to verify the privacy protection intensity of strategy compliance proof, and a temporary permission token which passes authorization is obtained; based on the temporary permission token, performing encryption calculation allowed by strategy compliance proof on the encrypted data block through homomorphic operation, and generating an encryption calculation result and a calculation correctness proof; carrying out joint cryptographic packaging on an encryption calculation result, a strategy compliance proof and a calculation correctness proof, and outputting security data through a verifiable calculation protocol; according to the invention, through a differential privacy budget real-time verification mechanism and homomorphic and cryptomorphic calculation based on the temporary permission token, dual guarantee of privacy protection and data security is realized.

A document is analyzed to identify attributes of the document. A set of statistical measures are calculated for each of the formatting attributes. A correlation between these statistical measures and elements of the document are identified. A hierarchical relationship is determined between the elements of the document. Data from the document is split into chunks using the hierarchical relationship. A representation of the document that includes the hierarchical structure is then generated. The representation is stored in a data store as a vector usable by a machine learning model to perform a query on the document according to the hierarchical structure.

A system for extracting a number of data elements from one or more unstructured data sources. The system may separate the text from the tables in a document, such that only the table data may be sent to the large language model (LLM), when the LLM only needs to review the table data. The system generates chunks from the document. The system associates unique identifiers with each chunk to provide traceability. The system identifies relevant chunks from the documents and includes the relevant chunks with a request to extract the data elements in a prompt to the LLM. The system also includes a request for the LLM to report the chunks used during extraction of the data elements. The reported chunks are stored with the extracted data for verification, auditing, and error control.

The invention discloses a data dynamic migration and storage method, device and equipment and a medium, and relates to the technical field of computers. The method comprises the steps that environment parameters of a current mechanical hard disk in a distributed cluster are collected; wherein the environment parameters comprise a vibration parameter, a temperature parameter and a noise parameter; quantifying the input and output delay performance of the current mechanical hard disk by using the environmental parameters of the current mechanical hard disk; if the input and output delay performance of the current mechanical hard disk is greater than a preset threshold value, searching a target mechanical hard disk combination in the distributed cluster; and migrating and storing the target data block of the current mechanical hard disk into the target mechanical hard disk combination based on an erasure code and a multi-copy mechanism. Through the scheme, the reliability of storing the data in the mechanical hard disk in the distributed cluster is remarkably improved.

The invention relates to the technical field of artificial intelligence chips, and provides a convolution operation method and device, electronic equipment and a storage medium, and the method comprises the steps: traversing convolution kernel elements, and determining a current to-be-loaded data block based on the shape of a target data block and the coordinates of the traversed current convolution kernel element block; based on the current to-be-loaded data block, determining a to-be-covered data block and a newly-added data block, and loading the newly-added data block into the shared memory; based on the initial position, reading the input data block from the shared memory, applying the input data block and the current convolution kernel element block, performing matrix multiply-accumulate operation, and accumulating an operation result to an output result; and determining a convolution operation result based on an output result obtained by accumulation after traversal is completed. According to the method and the device, only the newly added data blocks are loaded into the shared memory, so that repeated data loading can be avoided, the data volume loaded each time is reduced, and the memory bandwidth is saved.

The embodiment of the invention provides a memory recovery method and device, electronic equipment and a storage medium, and relates to the technical field of data storage. Under the condition that both the file system and the storage device support the perceptual garbage collection function, the device state of the storage device is obtained in response to the starting of the perceptual garbage collection function; according to the equipment state, determining whether an execution condition for sensing a garbage collection function is met or not; and if the execution condition of the garbage collection sensing function is met, the file system is linked to perform memory collection on the storage device, so that the file system can know the data blocks moved by the storage device, the file system is prevented from moving the data blocks moved by the storage device again, and the efficiency of the file system is improved. In addition, invalid blocks in the file system and the storage device can be synchronized, the situation that invalid data blocks in the file system are moved when the storage device executes memory recovery is avoided, and therefore the service life expenditure of the storage device and the write-in amplification value of the storage device are reduced.

The invention relates to the technical field of computer video processing, and discloses a low-delay video stream real-time processing method and device, and the method comprises the steps: obtaining original video stream data, and processing the original video stream data through employing a lightweight motion prediction method; processing the macro block data set and the predicted coding configuration parameter by adopting multi-thread assembly line coding to obtain a coded data block; establishing a data transmission mechanism to perform data flow control on the unified memory access interface; a heterogeneous task scheduling strategy is adopted to distribute task division results; a lightweight neural network is adopted to carry out parameter adaptive adjustment, and an optimized video stream processing result is obtained; according to the method, a zero-copy data transmission technology is adopted, and optimal configuration and efficient utilization of computing resources are achieved.

The invention discloses a distributed training method, device and equipment for a large-scale model, and the method comprises the steps: configuring a plurality of calculation nodes on current equipment according to the scale of a to-be-trained large model and a training task, and configuring the hardware resources and software resources of each calculation node, and obtaining a constructed distributed training environment; dividing a training data set and the to-be-trained large model by adopting parallel strategies of different dimensions to obtain a plurality of corresponding data blocks and a plurality of groups of model layers; starting a training process by loading model parameters, a plurality of data blocks and a plurality of groups of model layers to corresponding computing nodes in the distributed training environment; in the training process, the running state of each computing node is monitored, the training task of the corresponding computing node is dynamically adjusted by introducing an adaptive scheduling mechanism, and when a preset training termination condition is met, the trained target large model is stored. And the training requirements of high efficiency, stability and low cost can be met.

The invention provides a power distribution station house wireless communication method based on a cooperative coding algorithm, which comprises the following steps: data processing of dynamic environment adaptation: according to the real-time requirement of power distribution station house equipment data, segmenting original data into data blocks carrying priority labels, and dynamically generating a redundant coding strategy based on a real-time network state; network-aware cooperative transmission: transmitting data blocks and redundant information through multi-node cooperation, wherein a transmission path and a redundancy distribution proportion are dynamically adjusted according to network signal quality, node load and fault state; and fault-tolerant recovery of closed-loop feedback: the receiving end jointly decodes the data block and the redundant information, triggers a local retransmission instruction according to a decoding result, and feeds back the local retransmission instruction to the network architecture to optimize a subsequent transmission path.

The invention relates to the technical field of artificial intelligence, and provides a model performance optimization method, electronic equipment, a storage medium and a program product, and the method comprises the steps: obtaining a calculation operation corresponding to each model layer and a communication operation between the model layers based on a model structure; organizing all calculation operations and communication operations into a plurality of calculation and communication parallel units, wherein each unit comprises a first matrix multiply-accumulate operation, a fusion reduction operation and a second matrix multiply-accumulate operation; in each unit, input data is segmented so that a first matrix multiply-accumulate operation, a fusion reduction operation and a second matrix multiply-accumulate operation are performed in parallel based on different data blocks. According to the method, all calculation operations and communication operations are organized into a plurality of calculation and communication parallel units, and data are segmented in each unit, so that parallel execution of the calculation operations and the communication operations is realized, the degree of parallelism and the utilization rate of calculation resources and hardware resources are improved, and the model performance is remarkably improved.

Embodiments are directed to workload allocation for file system maintenance. A file system that includes storage nodes and snapshots may be provided such that each snapshot may be associated with a plurality of data blocks. If snapshots are deleted further actions may be performed, including: determining the dead blocks associated with the deleted snapshots such that each dead block may be a data block that may be unassociated with undeleted snapshots; adding the plurality of dead blocks to dead trees located on the storage nodes; determining an urgency score based on a workload model and file system metrics; determining delete tasks based on the urgency score; determining a portion of the storage nodes based on a number of delete tasks; and executing the delete tasks on the portion storage nodes to delete the dead blocks to return storage capacity to the file system.

Systems and methods disclosed herein are enabled to dynamically generate structured documents using one or more artificial intelligence models. A computing device receives an output generation request and uses a first AI model to retrieve data chunks from source documents and applicable templates. A second AI model ranks the retrieved chunks based on one or more metrics, such as vector similarity, keyword density, and temporal relevance. A third AI model subsequently generates a response using the ranked chunks, templates, and predefined operational boundaries for each chunk. The generated response is tagged with source identifiers to enable the traceability of the response back to corresponding chunks. The system transmits, via the computing device, the response, the retrieved chunks, and / or the source identifiers.

The invention discloses a multidirectional frame audio stream transmission method, device, equipment and medium, and the method is realized through cooperation of a transmitting end and a receiving end: the transmitting end cuts an original audio stream into independent audio frames, gives priority identifiers to the independent audio frames, and determines redundant coding parameters and transmission paths for different priority frames in combination with a predefined static strategy; generating a data packet containing an original data block and a redundant data block, and sending the data packet through at least one network path; and a receiving end caches the multi-path data packet, recovers lost data by using redundant data blocks to recombine a complete audio frame, and executes error concealment processing on the frame which cannot be recombined to generate a replacement frame. According to the method, based on a multi-path parallel transmission, forward error correction (FEC) redundancy mechanism and a cost-aware static scheduling strategy, lossless forwarding and instantaneous recovery of audio streams are realized on the premise of not waiting for network feedback and avoiding inter-frame dependence, and high-quality real-time audio transmission service can still be provided in a complex network environment.

The invention discloses a point cloud model rendering optimization method and system based on cloud edge collaboration, and belongs to the field of computer graphic processing, and the method comprises the steps: carrying out the spatial partitioning of multi-stage point cloud data, generating a point cloud hierarchical structure which can be quickly accessed, building a user portrait model, and predicting the region of interest of the user portrait model; the multi-level LOD point cloud data structure is distributed to an edge computing node, the current network state is evaluated in real time, and if the local cache of the edge node cannot meet the rendering requirement, the edge node adaptively requests for cloud incremental data; the client receives the data blocks from the edge nodes and uploads real-time interaction data to the edge nodes; and the client interaction record and the edge node cache hit rate are transmitted back to the cloud end, and the cloud end performs iterative optimization on a point cloud data distribution strategy by using a reinforcement learning model. According to the method, priority loading and high-precision rendering of the region of interest are achieved by constructing the multi-level LOD point cloud data structure, the delay and jamming phenomena are effectively avoided, and the user interaction experience is improved.

The invention discloses an AI model distribution and deployment system and method oriented to cloud edge collaboration, belongs to the technical field of artificial intelligence and edge computing crossing, and aims to solve the problems of low AI model distribution efficiency and poor deployment adaptability in a cloud edge collaboration scene. The method comprises the steps of collecting multi-dimensional attribute data of edge nodes and converting the data into multi-level capability tags, and triggering tag differentiation updating according to data change amplitude; constructing a model demand hierarchical description framework by relying on capability labels, converting distribution demands into quantitative query conditions, and adjusting and screening a target edge node set through a three-level progressive matching mechanism in combination with dynamic weight; coding data blocks corresponding to redundancy are generated based on the network state of a target node, adaptive cache nodes are screened by using a distributed cache mechanism, the data blocks are pre-stored, and a cache network is established and transmitted to the target node; according to the invention, efficient distribution and accurate deployment of the AI model in the cloud edge collaborative environment are realized, and the model transmission reliability and the node adaptability are improved.

The invention discloses a heterogeneous computing system, a cache consistency maintenance method and device, equipment and a medium, and relates to the technical field of heterogeneous computing. The method comprises the step of redefining a cache coherence protocol of the heterogeneous computing system as a metadata index protocol, a multicast synchronization protocol and a hybrid response protocol. Entropy weights of different feature dimensions are determined according to feature values of all data blocks of the heterogeneous computing system, and a protocol most matched with the task load running state at the current moment is determined from a metadata index protocol, a multicast synchronization protocol and a mixed response protocol according to the entropy weights of the different feature dimensions; and determining whether to switch the protocol according to whether the current protocol is consistent with the selected protocol. According to the method, the problem that the protocol cannot be effectively optimized in related technologies can be solved, and the consistency protocol which is most matched with the current task load of the heterogeneous computing system can be accurately determined.