3520 results about "Data chunk" patented technology

The invention relates to the technical field of information retrieval, and discloses a distributed storage and indexing method, which comprises the following steps of: dynamically fragmenting an original file through an improved consistent Hash algorithm to generate a plurality of data blocks with timestamps; constructing a three-dimensional Bloom filter index matrix containing timestamps, data types and content features for the data blocks with the timestamps, associating a B + tree local index with an inverted global index through a hierarchical index structure, and establishing an index update priority queue by adopting a two-channel synchronization mechanism, performing real-time increment synchronization on the hotspot index through a heartbeat mechanism, performing batch synchronization on the cold data layer index according to a cold data synchronization period, predicting a data distribution probability through a distributed query statistics probability table during query, and initiating multi-path query in parallel based on probability weight, and the query path is dynamically optimized according to the node group storage medium type and the inter-node network transmission delay, so that rapid distribution adjustment and access of distributed storage are realized.

A system for extracting a number of data elements from one or more unstructured data sources. The system may separate the text from the tables in a document, such that only the table data may be sent to the large language model (LLM), when the LLM only needs to review the table data. The system generates chunks from the document. The system associates unique identifiers with each chunk to provide traceability. The system identifies relevant chunks from the documents and includes the relevant chunks with a request to extract the data elements in a prompt to the LLM. The system also includes a request for the LLM to report the chunks used during extraction of the data elements. The reported chunks are stored with the extracted data for verification, auditing, and error control.

The invention relates to the technical field of computer video processing, and discloses a low-delay video stream real-time processing method and device, and the method comprises the steps: obtaining original video stream data, and processing the original video stream data through employing a lightweight motion prediction method; processing the macro block data set and the predicted coding configuration parameter by adopting multi-thread assembly line coding to obtain a coded data block; establishing a data transmission mechanism to perform data flow control on the unified memory access interface; a heterogeneous task scheduling strategy is adopted to distribute task division results; a lightweight neural network is adopted to carry out parameter adaptive adjustment, and an optimized video stream processing result is obtained; according to the method, a zero-copy data transmission technology is adopted, and optimal configuration and efficient utilization of computing resources are achieved.

Systems and methods disclosed herein are enabled to dynamically generate structured documents using one or more artificial intelligence models. A computing device receives an output generation request and uses a first AI model to retrieve data chunks from source documents and applicable templates. A second AI model ranks the retrieved chunks based on one or more metrics, such as vector similarity, keyword density, and temporal relevance. A third AI model subsequently generates a response using the ranked chunks, templates, and predefined operational boundaries for each chunk. The generated response is tagged with source identifiers to enable the traceability of the response back to corresponding chunks. The system transmits, via the computing device, the response, the retrieved chunks, and / or the source identifiers.

The invention discloses a multidirectional frame audio stream transmission method, device, equipment and medium, and the method is realized through cooperation of a transmitting end and a receiving end: the transmitting end cuts an original audio stream into independent audio frames, gives priority identifiers to the independent audio frames, and determines redundant coding parameters and transmission paths for different priority frames in combination with a predefined static strategy; generating a data packet containing an original data block and a redundant data block, and sending the data packet through at least one network path; and a receiving end caches the multi-path data packet, recovers lost data by using redundant data blocks to recombine a complete audio frame, and executes error concealment processing on the frame which cannot be recombined to generate a replacement frame. According to the method, based on a multi-path parallel transmission, forward error correction (FEC) redundancy mechanism and a cost-aware static scheduling strategy, lossless forwarding and instantaneous recovery of audio streams are realized on the premise of not waiting for network feedback and avoiding inter-frame dependence, and high-quality real-time audio transmission service can still be provided in a complex network environment.

The invention discloses a point cloud model rendering optimization method and system based on cloud edge collaboration, and belongs to the field of computer graphic processing, and the method comprises the steps: carrying out the spatial partitioning of multi-stage point cloud data, generating a point cloud hierarchical structure which can be quickly accessed, building a user portrait model, and predicting the region of interest of the user portrait model; the multi-level LOD point cloud data structure is distributed to an edge computing node, the current network state is evaluated in real time, and if the local cache of the edge node cannot meet the rendering requirement, the edge node adaptively requests for cloud incremental data; the client receives the data blocks from the edge nodes and uploads real-time interaction data to the edge nodes; and the client interaction record and the edge node cache hit rate are transmitted back to the cloud end, and the cloud end performs iterative optimization on a point cloud data distribution strategy by using a reinforcement learning model. According to the method, priority loading and high-precision rendering of the region of interest are achieved by constructing the multi-level LOD point cloud data structure, the delay and jamming phenomena are effectively avoided, and the user interaction experience is improved.

The invention discloses an AI model distribution and deployment system and method oriented to cloud edge collaboration, belongs to the technical field of artificial intelligence and edge computing crossing, and aims to solve the problems of low AI model distribution efficiency and poor deployment adaptability in a cloud edge collaboration scene. The method comprises the steps of collecting multi-dimensional attribute data of edge nodes and converting the data into multi-level capability tags, and triggering tag differentiation updating according to data change amplitude; constructing a model demand hierarchical description framework by relying on capability labels, converting distribution demands into quantitative query conditions, and adjusting and screening a target edge node set through a three-level progressive matching mechanism in combination with dynamic weight; coding data blocks corresponding to redundancy are generated based on the network state of a target node, adaptive cache nodes are screened by using a distributed cache mechanism, the data blocks are pre-stored, and a cache network is established and transmitted to the target node; according to the invention, efficient distribution and accurate deployment of the AI model in the cloud edge collaborative environment are realized, and the model transmission reliability and the node adaptability are improved.

The invention provides a parallel computing method and device, electronic equipment and a storage medium, and relates to the technical field of parallel computing, and the method comprises the steps: carrying out the first protocol operation of a target tensor based on each computing core in each stream processor cluster, and generating a data block containing the computing result of each computing core; writing a data block generated by each stream processor cluster into a shared cache; under the condition that each stream processor cluster completes the first protocol operation, reading a data block written by each stream processor cluster from the shared cache; and executing a second protocol operation on the data block read from the shared cache to generate a calculation result of the target tensor. According to the method and device provided by the invention, the parallel architecture and memory access characteristics of the artificial intelligence chip can be better matched, the unnecessary calculation delay and synchronization overhead of the cross-flow processor cluster in the parallel calculation process are reduced, the bandwidth utilization rate of the shared cache is improved, and the overall performance and calculation efficiency of parallel calculation are remarkably improved.

The systems and methods disclosed herein generate responses using data retrieved by validating vector embeddings using hash signatures. An output generation request is received via a computing device and includes an input that includes a content set and a command set. The content set includes data chunks, which refers to text, audio, image, and / or video data. A subset of the data chunks that fail to associate with existing hash signatures are selected. Each selected data chunk is validated against predefined constraints that define the operative boundaries of a guideline set and subsequently assigned a unique hash signature to indicate a degree of satisfaction of the data chunk with the guidelines. Using an artificial intelligence (AI) model, a response to the output generation request is generated in accordance with the data chunks, where each data chunk is associated with a corresponding unique hash signature.

The invention relates to the technical field of computer data processing, and discloses a data segmentation and query processing system under a distributed architecture. The system comprises a data segmentation module which divides data blocks based on a self-adaptive partitioning algorithm; the distributed storage module is used for distributing data blocks by using a consistent Hash algorithm and dynamically adjusting the distribution of virtual nodes; the index construction module is used for generating a global index table through a multi-layer inverted index algorithm; the query optimization module is used for converting query statements by utilizing a query rewriting algorithm; and the result aggregation module is used for processing the sub-query results by adopting a parallel aggregation algorithm. The system can dynamically process data according to data characteristics, optimizes the data storage and query process, effectively improves the data processing efficiency and query performance of a distributed system, and is suitable for scenes such as big data storage and analysis.

The invention relates to the technical field of distributed storage, in particular to a high-speed distributed storage system and method, and the system comprises a file index characterization module which carries out the statistics of the access frequency and the recent access timestamp of each data block in the system, and obtains the popularity magnitude of the file data blocks; according to the method, the popularity magnitude concept is introduced on the basis of linkage statistics of the data block access frequency and the recent access time, the data blocks with remarkable service influence in the current system can be recognized, the priority processing sequence of different data blocks in the repair process is divided, and the decision-making efficiency during fault recovery is improved. After the to-be-repaired data blocks are identified, the comprehensive weight values are formed through correlation calculation with the popularity magnitude, and all the weight values are arranged in a descending order, so that dynamic grading and sequential scheduling of repair tasks are realized, and important data blocks are guaranteed to preferentially obtain recovery resource support.

The invention provides an FPGA program loading and updating method and system, and relates to the technical field of FPGA configuration management. The FPGA program loading and updating method comprises the following steps: S1, based on an RS422 physical layer protocol, adopting a self-adaptive block verification algorithm, acquiring a signal-to-noise ratio parameter in real time through a channel quality monitoring module, dynamically adjusting the size of a data packet, and implementing a double-buffer alternate receiving mechanism; and verifying each data block by using cyclic redundancy check and serial number continuity detection, discarding abnormal data packets with bit error rates exceeding the limit, and generating a data block set passing verification. Through combination of an adaptive block check algorithm and channel quality dynamic monitoring, real-time optimization and adjustment of the size of a data packet in a transmission process are realized, transmission requirements in different channel environments are effectively adapted, the data retransmission probability is reduced, and through cooperative application of a double-buffer alternate receiving mechanism, cyclic redundancy check and a serial number tracking technology, the transmission efficiency is improved. And continuity and integrity of data receiving are ensured.

The invention relates to the technical field of data processing and communication, in particular to a multi-source heterogeneous data compression and transmission method and system for an intelligent fusion terminal, and the method comprises the steps: collecting multi-source heterogeneous data from different data sources through the intelligent fusion terminal, carrying out the data preprocessing of the collected multi-source heterogeneous data, and obtaining a data preprocessing result; comprising format unification and noise filtering; establishing a data feature analysis model, and performing feature extraction and classification on the preprocessed data; according to data feature extraction and classification results, dynamic weights are allocated to different types of data, and the data are divided into data blocks of different sizes according to the allocated dynamic weights and data features; for different types of data blocks, the priority of the data blocks is set, a proper compression algorithm is selected for compression, and the compressed data is transmitted through a network. The multi-source heterogeneous data is subjected to feature analysis and targeted compression processing, so that network bandwidth occupation and equipment energy consumption are reduced, and data transmission efficiency is improved.

The invention relates to the field of edge computing data security, in particular to an integrity auditing method and system in an edge computing environment. The problem that an existing cloud auditing model is high in delay and large in bandwidth consumption in an edge scene is mainly solved. A lightweight audit challenge is generated through random sampling, data block aggregation operation is executed at an edge node, Merkel Hash tree path evidence is extracted, and integrity proof of dual verification is formed; and the aggregated evidence and the path evidence are verified by using bilinear mapping, so that efficient and credible auditing is realized. Dynamic data updating and accurate positioning of damaged blocks are supported, communication overhead is remarkably reduced, and real-time credibility of edge data is guaranteed.

The invention relates to an application method of a GPU (Graphics Processing Unit) in design rule inspection in the field of integrated circuits, in particular to a layout design rule inspection method, system and equipment, a medium and a product. The layout design rule checking method comprises the following steps: inputting initial layout data, analyzing a design rule checking task to be executed, and marking the priority and complexity level of the task; the method comprises the following steps: dividing layout data into a plurality of data blocks according to regions on the basis of regional characteristics of the layout data, distributing tasks and corresponding data blocks to a GPU computing core according to task priorities and complexity levels, and distributing new data blocks according to the overall load condition of the GPU computing core after the GPU computing core completes data block processing, and repeating the steps until all tasks are executed, and outputting a layout design rule check result. The system, the computer equipment, the computer readable storage medium and the computer program product have the same beneficial effects as the layout design rule checking method.

A data processing system accesses a dataset from a data source and generates a set of embedding vectors representing the dataset in a latent space. The system splits the dataset into a set of data chunks and generates the embedding vectors. Each embedding vector represents a data chunk. The system may store the generated set of embedding vectors in a vector database that includes a plurality of embedding vectors. The system updates the embedding vectors by detecting a change to a first dataset that is represented by a first set of embedding vectors in the vector database, determining that the change to the first dataset is related to a first data chunk of the first set of data chunks included in the first dataset, updating a first embedding vector representing the first data chunk with the detected change; and storing the updated first embedding vector in the vector database.

The invention discloses an energy big data right confirmation method and system based on a hierarchical hash tree and dynamic authorization, and belongs to the field of energy big data management and information security, and the method comprises the steps: obtaining and standardizing original data in an energy scene, and dividing the original data into a data block set according to equipment and time; constructing a hierarchical hash tree and generating root hash; writing the root hash and the associated metadata into the block chain to realize right confirmation and evidence storage; executing proxy re-encryption according to the access token, and converting the ciphertext into a decryptable format; verifying the data consistency through the Hash path and the root Hash, and completing the access; and recording the access behavior and distributing transaction earnings by the smart contract based on the contribution degree. Hash calculation and local path updating of a data block level are supported through a layered Hash tree structure, so that when large-scale energy data is frequently updated, a new root Hash value can be quickly generated and right confirmation updating can be completed only by carrying out local Hash recalculation on a changed path.

A correlation-aware adaptive codebook compaction system for multi-modal data compression that preserves cross-modal relationships while providing enhanced reconstruction quality. The system analyzes temporal and spatial relationships between different data modalities to generate correlation maps that guide compression decisions. A virtual management layer performs stream characterization and adaptive routing, while a processing pipeline implements primary codebook compression with mismatch handling for novel data blocks. High-entropy data segments receive pre-compression processing before codebook compression. Sequential registration data is processed through matrix factorization and dedicated matrix codebooks. The system continuously monitors data distribution characteristics and automatically retrains codebooks when drift thresholds are exceeded. A neural upsampling subsystem uses correlation information to guide cross-modal enhancement processes through modality-specific networks and attention mechanisms. The unified output includes compressed data streams, correlation maps, synchronization metadata, neural model parameters, and updated codebooks, enabling synchronized reconstruction with preserved cross-modal relationships and enhanced quality through correlation-guided neural upsampling.

The invention belongs to the technical field of storage media, and particularly relates to a storage medium abrasion balancing system based on cold and hot data perception, which comprises a cold and hot data perception module used for monitoring the access frequency and the update time of data in real time so as to judge the cold and hot degree of the data, and establishing a heat table, recording a heat value of each data block or data page; and the data feature prediction module is used for predicting features of a future heat change trend and a data volume increase trend of the data by utilizing a machine learning algorithm according to historical data access and update records. According to the method, the hot and cold data sensing module is used for monitoring data access and updating conditions in real time to calculate a hot value, and the storage medium management module is combined to preferentially store hot data into blocks with less erasing times and store cold data into blocks with more erasing times. Therefore, the erasing times of the blocks of the storage medium tend to be balanced, and excessive abrasion of individual blocks is avoided.

The invention relates to the field of data storage, in particular to a storage and calculation separation multi-dimensional data block indexing method based on adaptive learning. Comprising the steps of receiving original multi-dimensional data and a query load, and outputting a low-dimensional key set through a first model; the low-dimensional key set and the historical query load generate a partitioning scheme through a second model, and the partitioning scheme performs physical recombination on the original multi-dimensional data to output a plurality of physical data blocks; constructing a learning type index through the physical data blocks and the corresponding partitioning schemes; identifying hot spot and cold spot areas through the query frequency of each node of the learning type index and the real-time query load continuous monitoring index; when the access mode changes, adjusting the index structure, and outputting a self-adaptive index; and quickly positioning and reading the target data block through the self-adaptive index to complete access. According to the method, the index of adaptive learning is provided, and the index structure and the physical data layout are adjusted, so that the query accuracy and efficiency in a complex scene are improved, and the data storage overhead is reduced.

The invention is suitable for the technical field of file management, and provides a file analysis method and system and a vehicle, the file analysis method and system are applied to a file analysis system, the file analysis system runs at a browser end, and the file analysis system comprises the steps that binary header information of a to-be-analyzed file is analyzed according to the byte position, obtained in advance, of the binary header information in the to-be-analyzed file, and the binary header information of the to-be-analyzed file is obtained; binary index position information of the to-be-analyzed file is obtained; according to the binary index position information, reading and analyzing binary index data of the to-be-analyzed file to obtain index data, determining a target data block meeting a preset requirement according to basic information of each data block, and reading the target data block; and analyzing the data in the read target data block to obtain formatted target data. According to the invention, the problem that the cost of cloud data storage and data analysis is increased due to the fact that a large amount of file data is analyzed and stored at the cloud in the prior art can be solved.

The invention belongs to the technical field of computer storage, and particularly relates to a solid state disk data management and efficient storage distribution method, which comprises a data characteristic sensing and classifying unit, a data characteristic sensing and classifying unit, a data processing unit, a data storage unit and a data distribution unit, wherein the data characteristic sensing and classifying unit is configured for analyzing write-in request data flow entering a solid state disk in real time or periodically, extracting various characteristic parameters associated with data blocks and storing the data blocks; classifying the data on the basis of the characteristic parameters, and distributing one or more data classification labels for each data block to be written or stored; and the self-adaptive storage allocation strategy unit is configured to be used for dynamically selecting an optimal physical block for data writing according to the data classification label and in combination with the physical block state information of the current flash memory medium. According to the invention, the solid state disk can better adapt to various dynamic and variable workloads, and especially shows more excellent continuous write-in performance, lower delay jitter and obviously prolonged service life when processing a large amount of random write-in, small block write-in and write amplification sensitive applications.

The invention discloses a storage management method and device of a multi-control storage system, a program product and a medium, and relates to the storage technology, the storage system comprises K controllers, K is a positive integer not less than 2, the storage management method of the multi-control storage system is applied to each controller, and the method comprises the following steps: receiving a write request, and splitting the write request into sub-requests; for any sub-request, judging whether the to-be-written data block corresponding to the sub-request belongs to the duplicated data block stored in the storage equipment or not through an affiliation controller of the sub-request; if not, writing the to-be-written data block into the storage device, and updating the metadata information based on the sub-request; and if yes, updating the metadata information based on the sub-request. By applying the scheme of the invention, the repeated data deletion can be realized in the multi-control storage system, and the K controllers can be utilized, so that the system performance can be effectively exerted, and the load balance can be ensured.

A video coder may code a flag indicating whether or not a neural network (NN)-based intra prediction mode is enabled before coding a decoder side intra mode derivation (DIMD) flag, and code a block of video data based on whether or not the NN-based intra prediction mode is enabled.

Systems and techniques are described for signing and verifying a data stream with an encryption signature. An example method includes determining, for a data block group, an initialization vector. The example method also includes generating, based at least in part on the data block group, encrypted hash data. The example method also includes determining an encryption signature based at least in part on the encrypted hash data, a private key, and the initialization vector. The example method also includes inserting the encryption signature into the data block group. The example method also includes causing transmission, by a network interface to a media server, of the data block group and the encryption signature. Finally, the example method includes causing verification, via the media server using a public key and synchronized data, that the data block group was generated by the computing device.

The invention discloses a data storage method and device and a computer storage medium, and the method comprises the steps: obtaining a system storage capacity, a new request resource capacity and a system used capacity, and obtaining a system capacity utilization rate based on the system storage capacity and the system used capacity; in response to the fact that the system capacity utilization rate reaches a threshold value and the current residual capacity does not meet the new request resource capacity, determining a storage capacity needing to be expanded based on the system storage capacity, the new request resource capacity, the used capacity and a preset target utilization rate, creating an expansion space corresponding to the storage capacity, and updating the system storage capacity; based on the updated system storage capacity, determining data to be stored of each node, and determining a data block to be migrated; migrating the to-be-migrated data block to the target node, and updating the data storage state of the current node; through the method, the change of the storage capacity can be dealt with more intelligently.

The invention relates to the technical field of big data analysis, and discloses a prospecting target area positioning method and system based on big data, and the method comprises the steps: collecting multi-source exploration data in real time through distributed nodes, completing coordinate normalization, semantic alignment and time synchronization through a spatial heterogeneous data flow engine, and generating a standardized incremental data block; performing local feature sensitivity analysis based on the historical model library, identifying a newly added feature dimension, and performing parameter increment updating by adopting a sliding window gradient descent method; inputting the updated model into a target evolution model driven by a Bayesian space-time probability field, and dynamically calculating the metallogenic probability of each space grid in combination with a stress field, an element migration path and historical verification data; and generating high, medium and low three-level target area maps according to probability sorting, and pushing the high, medium and low three-level target area maps to a three-dimensional visual decision terminal. According to the method, minute-level dynamic response of the target region under triggering of newly-added data is realized, computing resource consumption is reduced to be less than 5% of that of an original system, and prospecting efficiency and abnormal region identification timeliness are improved.

The invention discloses an intelligent completion code optimization method based on texture pre-rendering and multi-level cache, which comprises the following steps of: pre-establishing a grammar rule set and a GPU cache characteristic database, storing a grammar style sheet when a code editor is started, generating a font image set and loading current GPU cache parameters; after a candidate code segment list pops up through intelligent completion, dividing candidate code segments into data blocks, analyzing syntax types, determining character styles, and generating candidate code segment textures through instantiation rendering and a fragment shader; according to the current GPU cache parameter, the selected priority of the candidate code segment and the compressed volume, the texture is distributed to a first-level buffer, a second-level buffer or a global video memory according to a rule, and metadata is recorded and added into a global texture cache table; and after the user selects the candidate code segment, reading texture data from the corresponding GPU buffer, calculating a display coordinate by combining a cursor row and column index, an editor basic style parameter and a DPI scaling, completing texture rendering and drawing, and realizing optimized display of the intelligent completion code.

The invention relates to the technical field of multi-agent collaboration, in particular to a multi-agent collaboration method and system based on shared memory data exchange, and the method comprises the steps: creating and initializing a shared data area which is mapped and accessed by a plurality of agent processes in a physical memory; constructing an annular buffer structure on the shared data area, and realizing a lock-free read-write pointer propulsion mechanism based on atomic operation and a memory barrier so as to support a plurality of intelligent agents to perform data read-write concurrently; asynchronous notification of data updating is carried out between agent processes through a lightweight event notification mechanism; and each agent performs autonomous decision making and task scheduling based on the shared data so as to realize distributed collaboration. A data block version control and verification mechanism is introduced into the annular buffer structure, and when it is detected that the agents are abnormal, resources of the abnormal agents are automatically recycled, and a data consistency recovery process is triggered. Multiple times of data copying between a kernel mode and a user mode are eliminated, and communication delay is reduced.

Embodiments are directed to a file system that include object stores. An object store for write requests may be provided. Write ahead log (WAL) entries that include data blocks may be generated. A WAL object may be generated based on the WAL entries and stored in the object store. An in-memory overlay may be updated to associate the data blocks with the WAL object. A checkpoint operation may be executed to: generate an index object that includes index entries that associate other data blocks with data objects stored in the object store; update the index object to include index entries that associate the data blocks with the WAL object; store the updated index object in the object store; update the in-memory overlay to remove the association of the data blocks and the WAL object and update the in memory WAL to remove records of successfully checkpointed WAL objects.