2009 results about "Data chunk" patented technology

A data processing system accesses a dataset from a data source and generates a set of embedding vectors representing the dataset in a latent space. The system splits the dataset into a set of data chunks and generates the embedding vectors. Each embedding vector represents a data chunk. The system may store the generated set of embedding vectors in a vector database that includes a plurality of embedding vectors. The system updates the embedding vectors by detecting a change to a first dataset that is represented by a first set of embedding vectors in the vector database, determining that the change to the first dataset is related to a first data chunk of the first set of data chunks included in the first dataset, updating a first embedding vector representing the first data chunk with the detected change; and storing the updated first embedding vector in the vector database.

The invention belongs to the technical field of computer storage, and particularly relates to a solid state disk data management and efficient storage distribution method, which comprises a data characteristic sensing and classifying unit, a data characteristic sensing and classifying unit, a data processing unit, a data storage unit and a data distribution unit, wherein the data characteristic sensing and classifying unit is configured for analyzing write-in request data flow entering a solid state disk in real time or periodically, extracting various characteristic parameters associated with data blocks and storing the data blocks; classifying the data on the basis of the characteristic parameters, and distributing one or more data classification labels for each data block to be written or stored; and the self-adaptive storage allocation strategy unit is configured to be used for dynamically selecting an optimal physical block for data writing according to the data classification label and in combination with the physical block state information of the current flash memory medium. According to the invention, the solid state disk can better adapt to various dynamic and variable workloads, and especially shows more excellent continuous write-in performance, lower delay jitter and obviously prolonged service life when processing a large amount of random write-in, small block write-in and write amplification sensitive applications.

The invention relates to the technical field of multi-agent collaboration, in particular to a multi-agent collaboration method and system based on shared memory data exchange, and the method comprises the steps: creating and initializing a shared data area which is mapped and accessed by a plurality of agent processes in a physical memory; constructing an annular buffer structure on the shared data area, and realizing a lock-free read-write pointer propulsion mechanism based on atomic operation and a memory barrier so as to support a plurality of intelligent agents to perform data read-write concurrently; asynchronous notification of data updating is carried out between agent processes through a lightweight event notification mechanism; and each agent performs autonomous decision making and task scheduling based on the shared data so as to realize distributed collaboration. A data block version control and verification mechanism is introduced into the annular buffer structure, and when it is detected that the agents are abnormal, resources of the abnormal agents are automatically recycled, and a data consistency recovery process is triggered. Multiple times of data copying between a kernel mode and a user mode are eliminated, and communication delay is reduced.

The invention discloses a power system data integrity verification system based on a dynamic data block division mechanism and a block chain. The system comprises a data preprocessing module, a hash generation module, a block chain storage module and a verification processing module. The data preprocessing module divides original data into data blocks with dynamically adjustable sizes; the Hash generation module adopts an iterative Hash algorithm to generate a chain Hash structure and an integrity Hash abstract; the block chain storage module uploads the hash data to a block chain network through a smart contract; and the verification processing module outputs a verification result by recalculating and comparing the hash values. The system realizes efficient and reliable data integrity verification, and has the advantages of high storage efficiency, good verification precision and low system load.

The invention provides a Hadoop cluster incremental backup recovery method based on data block optimization, and relates to the technical field of data management, and the method comprises the following steps: obtaining distribution state information and historical change records of data blocks in a distributed file system, extracting space-time correlation characteristics, and constructing a data block dependency graph containing dependency direction and dependency intensity; identifying the changed data blocks, performing topological sorting, and identifying key path data blocks and independent data block clusters; calculating a distribution entropy value according to a copy distribution position, and determining a backup granularity type; a backup execution sequence is generated according to a topological sorting result, key path data blocks are backed up preferentially, independent data block clusters are backed up in parallel, and backup metadata containing dependency relationship identifiers and distribution entropy values are generated; and performing dependency consistency verification after recovery based on the dependency relationship identifier. According to the method, the backup recovery efficiency is improved, the data consistency is guaranteed, and the storage overhead is reduced.

The invention relates to the technical field of hard disks, in particular to a data protection method of a solid state disk and the solid state disk, the method comprises the steps that a flash memory medium is divided into a data block area, a log block area and a metadata area, and the metadata area comprises a mapping directory, a mapping block and a check point block; a block mapping table and a log page mapping table are established in a volatile memory, and dbMSN and lbMSN are stored. Adding write-in data to a log block page, updating a log page mapping table, and generating check points according to full writing of a log area. When no free page exists, the damaged log blocks are selected in a rotating mode to be combined, new data block migration pages are distributed, mapping is updated, the MSN is progressively increased, changes are written into the mapping blocks, and the mapping catalogue is updated when the mapping blocks are addressed. After restarting, positioning a mapping block by a mapping directory to recover block mapping, reading check points in a reverse order, and traversing an uncovered log area to reconstruct log page mapping; and comparing the dbMSN with the lbMSN, setting inconsistent log pages to be invalid, establishing consistent access and shielding invalid pages. According to the method and the device, the problems of data consistency and availability caused by metadata loss, uncertain merging state and low recovery efficiency of the hybrid mapping FTL in a crash scene can be solved.

The invention relates to a simulation result compression method and system based on semantic segmentation, equipment and a medium. The method comprises the following steps: performing feature extraction on a simulation result file to obtain semantic features; generating a segmentation point sequence according to the features by using a semantic segmentation model, and segmenting the file into a plurality of semantic blocks; extracting a compressibility feature vector for each block, and dynamically distributing an optimal compression strategy for the block through a prediction model; compressing each block by using the distributed strategy to generate a compressed data block; and finally, constructing a block sensing index based on the block type and the used strategy, and integrating the index and the data block to generate a compressed file. By adopting the method, the defects of limited compression ratio and incapability of supporting rapid random access caused by lack of semantic understanding of a general compression algorithm can be overcome, and by understanding the internal logic structure of the file and applying differential compression, the rapid retrieval and analysis capability of specific data is kept while the high compression ratio is realized.

The invention relates to a cache control method and a cache control device. The method comprises the following steps: receiving a data packet sent by at least one data source; the data packet comprises a command part and a data part; storing the command part into a command queue corresponding to the corresponding data source; splitting the data part of each data packet into at least two data groups, and storing data blocks in each data group into corresponding data queues; each data queue comprises data blocks in at least two data packets sent by the same data source, and the data blocks in the at least two data packets are spliced; and storing the commands in the command queue and the data blocks in each data queue into a memory based on the transmission channel. By adopting the method, the data transmission efficiency can be improved.

The invention relates to a remote sensing image data processing method and system based on a Spark architecture, and belongs to the technical field of remote sensing image data processing. The method comprises the following steps: converting a collected original file into a COG format, constructing a hybrid index through a spatial dimension reduction technology, extracting metadata and establishing a metadatabase; based on the hybrid index, quickly positioning a target tile through spatial query, reading a data block, and constructing a TileLayerRDD of a spatial partition in a Spark cluster; and dynamically assembling and calculating the directed acyclic graph according to the algorithm warehouse template, and submitting the directed acyclic graph to a Spark cluster for distributed memory calculation. According to the method, rapid retrieval is achieved through the COG format and the mixed index, pre-slicing is replaced with Spark distributed computing, the storage pressure is greatly reduced, and on-demand real-time analysis and high-concurrency online service of mass remote sensing data are finally achieved through micro-service and elastic resource scheduling.

The invention relates to a bearing fault diagnosis method based on variational mode decomposition and time sequence partitioning cross attention fusion, which comprises the following steps: acquiring an original vibration acceleration signal of a rolling bearing, and constructing a standardized original data set; segmenting the standardized original data into a plurality of data blocks, and generating a time domain embedding feature; based on the time domain embedded features, extracting high-order global time domain features by using a multi-head self-attention mechanism, residual connection and a feedforward neural network; decomposing the standardized original data into a plurality of intrinsic mode functions, and extracting frequency domain distribution features through a convolutional neural network; taking the frequency domain distribution characteristics as query vectors, retrieving and matching related fault context information in the global time sequence characteristics, realizing weighted fusion of time-frequency modes, inputting fused fault representation vectors into a classifier, and calculating a result of a bearing health state; and constructing a loss function containing label smoothing and a dynamic learning rate scheduling strategy, and carrying out iterative optimization on model parameters until the model converges.

The invention discloses a single-file concurrent read-write method and system of a distributed file system. The method comprises the steps that S1, a conservative write range lock application request of a preset interval is sent to a metadata server; the range lock takes a block as the minimum granularity; s2, segmenting to-be-written data into a plurality of data blocks according to a preset block length, and applying for a corresponding locking interval in a local memory lock table according to a starting position and an ending position of each data block in a target file; judging whether the currently applied locking interval is overlapped with the registered locking interval or not; s3, if the data block is not in the write range lock, applying for an interval for expanding the write range lock from the metadata server; s4, distributing the segmented data block write-in requests to block engines of corresponding storage nodes, so that the block engines add data block mutual exclusion locks at corresponding positions; and S5, in response to a request of closing or refreshing the target file by the user, sending a request of releasing the write range lock interval. According to the method and the device, the performance is not reduced on the premise of improving the strong consistency.

The invention discloses a method and equipment for compressing multi-dimensional data based on column storage and self-adaption, and the method comprises the steps: recombining original multi-dimensional data into a column data set structure, and organizing data block storage for the column data set structure according to a hierarchical structure; each data block stores observation grid data of a single variable of multi-dimensional data at a certain moment so as to support column type storage and self-adaptive compression; and carrying out real-time statistical analysis on characteristics or parameters of variables in the data blocks, and dynamically selecting and configuring a filter and parameters thereof to compress the multi-dimensional data. The data of the same variable are stored together through column storage, so that the data access efficiency is greatly improved, especially during variable-level analysis. Meanwhile, the self-adaptive compression technology can dynamically select a compression mode according to the distribution characteristics of the data, so that the compression rate of the data is effectively improved, and the occupied storage space is reduced. The processing speed of the meteorological grid point data can be obviously improved, and the storage cost is reduced.

The invention provides an intelligent grading and excess subscription management system and method for a GPU video memory, and relates to the technical field of GPU video memory management, and the method comprises the steps: firstly building a grading storage system comprising a first performance region and a second performance region; then receiving a video memory allocation request containing task priority and service quality requirements, and monitoring a data access mode and access frequency; the initial placement position of the data block is dynamically determined through the intelligent engine according to the task priority, the data access mode and the frequency; predicting target access probability distribution of each data block by adopting an LSTM network; then migrating data between the first performance area and the second performance area through a swap-in and swap-out mechanism based on the distribution and periodicity characteristics, and ensuring the service quality of the first performance area; finally, the global oversale proportion is dynamically adjusted through an overpurchase safety management module, an independent virtual address space is distributed for each task, and safe overpurchase can be achieved through the process.

The invention relates to the technical field of visual rendering, and provides a visual rendering engine optimization method and system. The method comprises the following steps: acquiring to-be-processed data, performing data partitioning, and constructing a spatial index for the partitioned data; monitoring scrolling, zooming and translation events of the canvas; when it is monitored that the view change meets a preset condition, calculating a latest view state, packaging the latest view state into a message object, and configuring a priority identifier for the message object; according to the message object transmitted by the main thread and received by the Worker, querying a spatial index of a Worker memory, positioning all data blocks intersected with or contained in the current view, and sampling and aggregating to obtain a vertex data array; and after the main thread receives the vertex data array sent back by the Worker, the vertex data array is transmitted into a WebGL buffer area for rendering.

The invention relates to the technical field of distributed storage, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a data storage method, device, equipment and medium based on partition identifier mapping, and the method comprises the following steps: receiving a file and dividing the file into data blocks to generate block identifiers; generating a partition identifier based on the file identifier and the block identifier through hash mapping; creating a partition disk pack mapping table and writing an initial relationship; querying the mapping table to obtain a disk group, and writing the data block into a physical disk to generate a copy; monitoring the health of the disk, keeping the partition identifier and the block identifier unchanged when a fault occurs, and updating the mapping relation to a new disk group; and receiving a read-write request of the target partition identifier, querying the mapping table to obtain the target disk pack, and executing access. According to the method, the partition identification and the block identification are kept unchanged, fast fault tolerance is achieved only by updating the mapping relation, metadata updating expenditure is reduced, bottom layer change is shielded through partition identification routing, and access transparency and high availability are achieved.

According to the large file uploading method and device, the electronic equipment and the storage medium provided by the invention, the operation environment parameters of the target client are collected, and the file uploading strategy containing the data block size, the concurrent transmission quantity and the retry mechanism is dynamically generated based on the parameters; meanwhile, according to the strategy, the file to be uploaded is divided into a plurality of data blocks and integrity verification is executed, the uploading state of each data block is persistently stored in the uploading process, and uploading of the uncompleted data blocks is recovered based on the uploading state after transmission interruption is detected. The problems that due to the fact that a fixed fragment size and a static concurrent strategy cannot adapt to different device performance and network environments, the uploading state is lost after page refreshing or browser restarting is caused by depending on browser cache, the verification efficiency is low, and uploading fails, resource utilization is not good, and uploading starting is delayed can be solved. The technical effects of improving the environmental adaptability, the resource utilization efficiency and the uploading success rate of Web cloud disk large file uploading are achieved.

The invention discloses a public opinion home address standardization method and system, and the method comprises the following steps: S1, obtaining and processing multi-source address original data to obtain an address data block, processing the data block based on a pre-trained text embedding model to obtain a corresponding semantic vector, and storing the address data block and the corresponding semantic vector, establishing a multi-source address knowledge base; s2, processing public opinion text data through the text embedding model to obtain a query vector of the public opinion text data; s3, querying the multi-source address knowledge base based on the query vector to obtain a plurality of address data blocks related to the query vector; and S4, constructing a cue word template based on the public opinion text data and the address data blocks, and inputting the cue word template into a large language model to perform public opinion attribution address standardization analysis.

The invention discloses a distributed file backup method for a sonar database system, which belongs to the technical field of sonar data processing and comprises the following steps: acquiring file data containing underwater acoustic signal data from the sonar database system; fragmenting the file data to generate data blocks and dynamically adjusting the sizes of the data blocks; generating a unique identifier for the data block based on the content hash value and the metadata; selecting distributed storage nodes based on availability, storage capacity, network delay and geographical distribution factors; the transmission data block uses an encryption protocol; updating the storage position and state of the backup metadata record; periodically verifying the integrity and availability of the backup data; executing a data recovery process to recover the original file data; the dynamically adjusted backup policy is based on real-time operation requirements and data importance levels. According to the method, through distributed storage and dynamic optimization, the backup efficiency, the data security and the system reliability are improved, and the method is suitable for sonar data management in an underwater detection system.

The invention discloses a multipath relay transmission method and system for underground confined space construction data, and belongs to the technical field of wireless communication network routing, and the method comprises the steps: obtaining a signal quality parameter of a communication link and a distribution density parameter of an ad hoc network node, and combining the load state of the node and the information of a neighbor node, decision calculation is executed on a plurality of relay nodes in parallel, relay decision vectors corresponding to the relay decision vectors are generated, a plurality of candidate transmission paths are formed through evaluation, construction data blocks to be transmitted are fragmented and distributed to the corresponding optimal candidate transmission paths to be sent in parallel, all received data fragmented sub-blocks are verified and recombined, and the construction data blocks to be transmitted are obtained. And recovering a complete construction data block. A distributed relay decision based on network state feature fusion, data adaptive fragmentation based on path stability and a multipath parallel transmission and feedback optimization mechanism are adopted, so that the data transmission reliability in an underground complex environment can be improved, the transmission delay is reduced, and the dynamic optimization of network performance is realized.

The invention provides a structured memory data processing method and system oriented to long time sequence interaction, and is applied to the technical field of natural language processing and artificial intelligence memory modeling. The method comprises the following steps: acquiring a real-time interaction data stream, partitioning the real-time interaction data stream into dialogue data blocks, loading a hierarchical attribute mode and a previous time step attribute tree instance, inputting serialized texts of the hierarchical attribute mode and the previous time step attribute tree instance into a generative model, generating a writing, rewriting, deleting or null operation instruction aiming at a leaf node path, analyzing, updating, generating a current attribute tree instance, and storing the current attribute tree instance; by means of the scheme, lossy compression and structured evolution of the infinite long dialogue stream can be achieved, memory forgetting is relieved on the premise that a context window is not expanded, and long-time-sequence information retrieval precision and storage efficiency are improved.

The invention provides a gas sensor compliance adjustment detection method and system based on a double rule engine, and relates to the technical field of sensor detection.The method comprises the steps that backtracking segmentation is conducted, and M pieces of time sequence concentration data are obtained; carrying out monomer-level adjustment compliance verification, and outputting an adjustment anchor point stamp sequence; performing cross-data-source space-time alignment segmentation to obtain a plurality of space-time associated data blocks; constructing a cross check relation topology; executing space-level adjustment compliance dynamic verification, and outputting a plurality of space compliance factors; performing adjustment compliance conflict judgment, and outputting a plurality of dynamic risk entropies; performing risk attenuation weighting, and outputting a real-time health degree index; and mapping and outputting a real-time gas sensing risk level. The technical problems that in the prior art, due to the fact that the source of gas concentration fluctuation cannot be accurately recognized by depending on a general anomaly detection algorithm or a manual checking means, the illegal behavior that a sensor is not adjusted according to regulations cannot be accurately recognized, and the safety and reliability of gas monitoring are affected are solved.

The embodiment of the invention provides a data transmission method and device in a computing system, electronic equipment and a storage medium, and the method comprises the steps: firstly, in response to a block-level data transmission instruction initiated by a meta-thread running in a first computing component, determining target data block information of the block-level data transmission instruction, generating transmission information pointing to the second computing component based on the target data block information; the block-level data transmission instruction is used for representing data transmission between the first computing component and the second computing component, and the operation of the meta-thread is independent of that of the common working thread; then, the transmission information is sent to the second computing component, so that a target data block represented by the transmission information is transmitted between the first computing component and the second computing component, and a block-level data transmission instruction is initiated by utilizing a meta-thread which runs in the first computing component and is independent of a common working thread. Data transmission control and hardware resource decoupling of dense calculation tasks are realized, and the resource utilization rate and the parallel cooperation efficiency are improved.

The invention relates to the technical field of data cleaning, in particular to an automatic data cleaning method and system based on a large language model.The method comprises the steps that original unstructured text data to be cleaned is preprocessed to generate standardized data blocks, and vector indexes are established for the data blocks; performing error detection on the data block based on a retrieval enhancement generation technology; in response to an error detection result, calling a large language model to generate a plurality of candidate repair schemes for the identified errors, and performing score sorting on the candidate repair schemes based on the multi-dimensional features; carrying out comprehensive confidence evaluation on the sorted candidate repair schemes, and executing a grading decision according to an evaluation result; and storing the result of the grading decision, and establishing a mapping relationship between the repair result and the original data to form a traceable data cleaning version record. Quantitative evaluation and three-level automatic decision making of the repairing result are achieved, and the problem that the result is uncontrollable due to the fact that LLM is directly used is solved.

An AI based system and method for processing and querying files. A method of processing files for an artificial intelligence (AI) querying service includes: hierarchically parsing the files into a set of hierarchically connected data chunks; generating metadata for each of the hierarchically connected data chunks, wherein the metadata includes hierarchical information; processing the hierarchically connected data chunks and metadata with an embedding model to generate vector embeddings that include the hierarchical information; generating textual summaries from the hierarchically connected data chunks; and storing the vector embeddings, textual summaries, and hierarchically connected data chunks for the AI querying service.

The invention relates to the field of data processing, in particular to an information retrieval method for a smart city system, which comprises the following steps of: obtaining a historical data block of a business system and user interaction session data, and performing word segmentation to obtain an original word and a to-be-processed word set; calculating topic probabilities and topic correlation degrees of original words and historical data blocks through a topic model, screening candidate words and calculating importance degrees of the candidate words; expansion words and historical perception words are generated and spliced into a dynamic sequence, the semantic similarity of the sequence and historical data blocks is calculated, a preset number of data blocks are sequenced and displayed, and retrieval is completed. The topic probability of the original word and the historical data block and the corresponding topic correlation degree are calculated through the topic model, the candidate word inherits the topic attribute of the original word, the topic probability distribution consistency of the candidate word and the original word is quantified in combination with the JS divergence to screen the extension word, the topic deviation risk of semantic extension is avoided from the source, and the semantic extension efficiency is improved. And the retrieval matching precision is improved.

A method for managing a stream includes: intercepting a request originated from a streaming storage system that specifies a data tiering operation; analyzing the request based on a policy to obtain an analysis result; identifying, based on the analysis result and function metadata, a first function and a second function that need to process data chunks of the stream before performing the operation; making, based on the identifying, a determination that the first function and second function are available internally; processing, using the first function and based on the policy, the data chunks to obtain the processed data chunks; identifying, using the second function, that the processed data chunks need to be tiered to a first long-term storage; tiering the processed data chunks to the storage; and notifying, once the processed data chunks are tiered to the storage, the streaming storage system that the data tiering operation is completed.

The present disclosure relates to systems, non-transitory computer-readable media, and methods for generating block content elements from meeting data of multiple video calls to add to data blocks of a virtual space. In particular, in one or more embodiments, the discloses systems utilize a large language model to process meeting data across multiple video calls and generate the block content elements for data blocks. For example, in some embodiments, the disclosed systems generate new block content elements based on detecting meeting data from additional video calls. Further, in one or more embodiments, the disclosed systems generate block content elements according to block type of a data block and extract meeting data to generate summaries, action items, document elements, or dates. Moreover, in some embodiments, the disclosed systems provide options and prompts to generate or update a virtual space using meeting date from video calls.

The invention relates to the technical field of data synchronization, in particular to an incremental synchronization and verification method and system for an access control authorization list, and the method comprises the steps: initializing a terminal and a server; the server side obtains the list change data to progressively increase the version number, and pushes the version number to the terminal equipment; the terminal initiates an incremental request to the server according to the notification message and obtains a server response incremental data block; the terminal performs multi-factor data verification on the incremental data block; performing cyclic synchronization on the incremental data; the terminal performs full-amount consistency verification on the incremental data; server real-time message notification and long connection transmission are adopted to replace a traditional polling mechanism, and the millisecond-level response speed is achieved; the rpcx protocol is matched with a proto binary format to perform data transmission, so that the protocol header overhead is reduced, and the serialization efficiency is improved; a multi-factor data verification mechanism is introduced, list number verification, incremental version number continuity verification and full-text hash verification are combined, a multi-level data consistency guarantee system is constructed, and the problems of data missing and wrong sending are avoided.

In some embodiments, a system transforms an initial user query into a first rewritten query using a first query rewriting algorithm, executes a search of a data repository using the first rewritten query to generate a set of results, executes a chunking process on the set of results to generate chunks of data, transforms the initial user query into a second rewritten query using a second query rewriting algorithm, generates corresponding embeddings for the second rewritten query and the chunks of data using a reranking model, selects a subset of the chunks of data based on a comparison of the embeddings for the chunks of data and the embedding for the initial user query, generates a prompt based on the initial user query and the subset of the chunks of data, submits the prompt to a Large Language Model (LLM) to generate a response to the initial user query.

The disclosure relates to a generative AI (artificial intelligence)-based meeting minutes generation method and a computing device using the same, and a generative AI-based meeting minutes generation method using a computing device according to an embodiment of the disclosure may include: collecting transcript text generated during a meeting to generate respective data chunks for every configured unit; summarizing the transcript text included in the data chunks to generate interim summary data for each data chunk using a large language model (LLM); grouping the data chunks by reference number to generate multiple topic groups and generating topic text encompassing the interim summary data for each topic group using the large language model; and inputting the topic text and the interim summary data for each topic group into the large language model to generate topic-wise summary data obtained by summarizing the interim summary data according to the topic text.