36 results about "Distributed retrieval" patented technology

The invention relates to the technical field of data processing, in particular to an intelligent document retrieval and generation system based on metadata driving, which comprises a metadata extraction module, a dynamic index construction module, an intelligent routing module, a distributed retrieval module and a result generation module. The metadata extraction module extracts organization names, timestamps and technical classification tags from an input document, maps the organization names, the timestamps and the technical classification tags into structured key value pairs, and transmits the structured key value pairs to the dynamic index construction module through a streaming pipeline; and the dynamic index construction module constructs a skip list index based on discrete fields to realize rapid matching, fuses metadata features and document semantics by adopting a multi-head attention mechanism, and outputs a structured response through topic clustering. According to the method, through a metadata pre-filtering, path optimization and semantic analysis cooperation mechanism, the multi-dimensional composite query efficiency in a distributed environment is improved, and noise interference is suppressed.

The invention provides an airfoil profile aerodynamic performance prediction method based on a large model retrieval enhancement generation framework, and the method comprises the steps: building a weakly-coupled database through fusing the geometric and aerodynamic characteristics of an airfoil profile, further building a strongly-coupled airfoil profile data distributed retrieval model, building the mutual cooperation of two stages of rough arrangement and fine arrangement, and achieving the prediction of the aerodynamic performance of the airfoil profile. And various characteristics of the airfoil profile are comprehensively considered, so that data can be retrieved more accurately. Meanwhile, fine-grained airfoil aerodynamic knowledge is generated in combination with a large-parameter language model, and the knowledge is optimized through knowledge distillation and fine tuning technologies. According to the method, explainable airfoil aerodynamic design priori knowledge can be excavated, rapid intelligent design of airfoils is powerfully supported, and the method has important significance in promoting the airfoil aerodynamic design to develop towards the intelligent and efficient direction.

The invention relates to a Chinese language and literature computer networking query reading system. According to the system, a query request processing module carries out mixed feature analysis processing according to a query request of a user, and a mixed query vector fusing radicals and semantics is generated; a distributed retrieval module performs distributed retrieval in a preset space-time fragment index structure based on the mixed query vector to obtain a candidate literature data set; the preliminary retrieval optimization module is used for carrying out self-adaptive conversion processing on abnormal characters and traditional and simple characters in the candidate literature data set according to a Chinese character evolution rule and a context to obtain standardized text data; and the visual query output module performs knowledge graph enhancement processing on the standardized text data to generate a visual query result. According to the system, efficient query and visual display of Chinese language and literature works are achieved through multi-modal processing, multi-dimensional indexing and knowledge graph technologies, and the problems that a traditional query mode is single in input, limited in retrieval dimension, insufficient in text processing and the like are effectively solved.

The invention discloses a communication data intelligent retrieval method and platform based on semantic analysis, and relates to the technical field of natural language processing, and the method comprises the following steps: after receiving an original query text of a user, binding an association retrieval permission strategy based on a user ID, loading a user portrait to complement semantics to obtain a plurality of semantic equivalence complement statements, and storing the semantic equivalence complement statements in a database; the method comprises the following steps: extracting feature words of communication data, constructing a plurality of hierarchical retrieval feature trees in combination with a domain knowledge graph, then matching a distributed retrieval unit, carrying out dynamic superposition retrieval by taking a retrieval permission strategy as a constraint to obtain a plurality of refined result sets, carrying out descending order arrangement and integration on the refined result sets according to metadata timestamps, and outputting a communication data time series retrieval sequence. According to the method and the device, the technical problems of low retrieval efficiency and difficulty in accurately positioning the communication data caused by single semantic understanding in traditional communication data retrieval are solved, and the technical effects of accurate semantic understanding of the communication data, orderly and efficient retrieval and accurate positioning of the communication data meeting the intention of the user are achieved.

The disclosure describes an enhanced distributed retrieval augmented generation (RAG) system. The enhanced distributed RAG system provides RAG data to a local small foundation system to form prompts. The enhanced distributed RAG system creates local query optimized representations (QORs) that are matched with queries sent to the local small foundation system to create prompts. The enhanced distributed RAG system is located on one edge site and is connected to the global federated RAG module. The global federated RAG system is connected to other enhanced distributed RAG systems in other edge sites. QORs can be distributed across the enhanced distributed RAG systems and the global federated RAG module by promoting local QORs to global QORs and vice versa.

The invention relates to a retrieval routing optimization method and device based on a distributed retrieval system, and relates to the field of computer technology distributed retrieval, the method comprises the following steps: obtaining inverted metadata of a document, including routing information, fragmentation information and a document storage identifier, judging whether the document is a new document, and combining the resource state and availability of each node to optimize the retrieval routing optimization of the document. And determining an optimal routing path by adopting an optimal execution fragmentation formula. A retrieval request is split into a plurality of sub-requests and is directionally issued to a target fragment, and the coordination node integrates return results of the fragments and then returns the return results to the client. The device comprises an information integration module, an inverted metadata recorder and a retrieval routing control module, and supports a dynamic configuration routing strategy, a document state refreshing mechanism and performance monitoring parameters. By introducing the document state identifier, the dynamic routing algorithm and the node resource monitoring mechanism, accurate routing and efficient processing of the retrieval request are realized, so that the retrieval performance and the resource utilization rate of the system are improved.

The invention relates to the field of internet search service, and discloses a big data-based investment attraction clue analysis system, which comprises a query analysis unit, a query analysis unit, a query analysis unit, a query analysis unit and a query analysis unit, wherein the query analysis unit calls an event sequence template for defining a meta event type and a standard relative time window constraint interval; the distributed retrieval and aggregation unit is used for retrieving and aggregating meta event types to entity objects; the sequential logic verification unit is used for counting the global document frequency of the meta event, reversely mapping the global document frequency to generate a zoom coefficient, dynamically correcting a time window, and judging that only an entity object in a dynamic verification interval after correction is passed; the object sorting unit generates sorting scores according to template matching integrity, and by constructing a density and window reverse coupling mechanism, the sequential logic boundary is adaptively compressed in a high background noise environment, and the retrieval signal-to-noise ratio and the logic confidence coefficient are maintained.

The invention discloses an artificial intelligence large model training process heterogeneous resource index time sequence visualization method and system based on meteorological supercomputing, and the method comprises the following steps: P1, collecting supercomputing architecture intelligent computing resource indexes, and generating a log file in a standard text format; step P2, analyzing the log and storing the log in a database, and storing the log in a distributed retrieval and analysis engine; p3, acquiring training task operation information, and storing the training task operation information in a distributed retrieval and analysis engine; step P4, performing index information association and aggregation to form resource index time sequence information of the computing nodes during operation of the large-model training task, and issuing the resource index time sequence information as a standard API retrieval interface; and step P5, visually displaying the resource index time sequence. According to the method, the problems that the resource indexes of the large model training task based on supercomputing are difficult to monitor and the resource indexes and the operation are dynamically associated can be solved.

The invention discloses a high-dimensional data retrieval method based on distributed computing and mixed indexing. The method comprises the following steps: step 1, data preprocessing; step 2, cluster generation; step 3, index construction; step 4, consistent Hash: distributing each cluster to a virtual node of a Hash ring, dynamically adjusting data distribution on the virtual node according to a load strategy, and mapping the virtual node to a physical server; and step 5, hierarchical aggregation: mapping a query request to a node and adjacent nodes on a hash ring through a hash function, performing candidate set calculation in parallel, merging candidate sets sent by all the nodes by a Spark center node, and finally optimizing a result. According to the method, high-recall-rate and low-delay distributed retrieval is achieved through mixed index construction, dynamic consistency hash allocation and hierarchical aggregation strategies, meanwhile, system load balancing and index updating efficiency are guaranteed, and the method is suitable for large-scale high-dimensional data scenes.

The present application relates to the technical field of data processing, and more particularly to an intelligent document retrieval and generation system based on metadata driving, comprising a metadata extraction module, a dynamic index construction module, an intelligent routing module, a distributed retrieval module and a result generation module. The metadata extraction module extracts the agency name, timestamp and technical classification label from the input document and maps them into structured key-value pairs, which are transmitted to the dynamic index construction module through a streaming pipeline. The dynamic index construction module constructs a skip list index based on discrete fields to achieve fast matching, adopts a multi-head attention mechanism to fuse metadata features and document semantics, and outputs structured responses through topic clustering. The present application improves the efficiency of multi-dimensional compound queries in a distributed environment and suppresses noise interference through a metadata pre-filtering, path optimization and semantic analysis collaborative mechanism.

The invention discloses a data query method, system and equipment based on distributed retrieval enhancement generation, and belongs to the field of data processing. The method comprises the following steps: a first node receives a query request initiated by a user through a gateway; the first node sends a query request to at least one second node through the gateway; the first node and the second node retrieve own knowledge bases respectively to obtain candidate documents matched with the query request, and generate retrieval results based on the candidate documents through own gateways; the second node sends a retrieval result generated by the second node to the first node through the gateway; the first node reorders the retrieval result generated by the first node and the retrieval result sent by the at least one second node through the gateway, and determines a target retrieval result based on the reordering result; and the first node generates a query result corresponding to the query request based on the target retrieval result through the large model. According to the embodiment of the invention, the information coverage rate and the scene adaptability can be improved, so that the accuracy of the query result is improved.

The present invention discloses a blockchain-based traffic data access control system, which relates to the technical field of traffic data management. The components of this system include: a data fragmentation storage module, a retrieval and recombination module, an access frequency monitoring module, a traffic load assessment module, and a resource allocation and frequency regulation module. By splitting and encrypting traffic data and storing it in multiple blockchain nodes, even if some nodes are maliciously attacked, it is difficult for attackers to obtain complete and valid data, effectively preventing the risks of data leakage and tampering, ensuring the integrity and confidentiality of data. At the same time, based on the immutable feature of the blockchain, the entire process of data access is reliably recorded, further enhancing the security and traceability of data. Additionally, by using smart contracts to drive distributed retrieval and recombination technologies, it ensures that users can accurately obtain complete data and effectively address the data integrity issue when some nodes are attacked.

The invention discloses a block chain traffic data access control system, and relates to the technical field of traffic data management, and the system comprises a data fragmentation storage module, a retrieval and recombination module, an access frequency monitoring module, a flow load evaluation module, and a resource allocation and frequency regulation and control module. According to the method, the traffic data is segmented and encrypted and stored in the multiple block chain nodes, even if part of the nodes are subjected to malicious attacks, an attacker is difficult to obtain complete and effective data, the data leakage and tampering risks are effectively prevented, the integrity and confidentiality of the data are ensured, and meanwhile, based on the non-tampering characteristic of the block chain, the security of the data is improved. The whole process of data access is reliably recorded, the security and traceability of the data are further enhanced, and an intelligent contract is utilized to drive a distributed retrieval and recombination technology, so that a user can accurately obtain complete data, and the problem of data integrity when part of nodes are attacked is effectively solved.

This invention discloses a lightweight distributed retrieval method and system based on Dubbo. The method includes: extracting unstructured vector features from image data captured and parsed by a device or video stream; storing the unstructured vector features in a feature writing node, storing them in a file with device ID and date as a partition; setting up a retrieval module including multiple retrieval nodes to provide unstructured vector feature retrieval capabilities; setting up a routing module; and setting up a distributed retrieval management module, which, when making a retrieval request, extends Dubbo's consistent hash routing mechanism, requests the retrieval scope according to device and time partitions, and sends the unit composed of device ID and day in the request to the retrieval node calculated by the corresponding route for multi-threaded requests, with each request retrieving the top N most similar items.

The invention discloses an index recovery method, device and equipment and a computer readable storage medium, which are applied to the technical field of computers, and the method comprises the following steps: when a fault occurs, analyzing a data topological structure corresponding to an index data fragment stored by a node; based on the fragment access log of the node, calculating a heat value of the index data fragment by adopting a time decay model; determining a loading thread count according to the node resource state; determining the priorities of the index data fragments based on the business query priorities, the popularity values and the fragment memory efficiency, and determining a loading sequence according to the priorities; and according to the loading thread count and the loading sequence, loading the index data fragments in the nodes from the persistent storage to the cache, and constructing a memory finite state converter structure capable of being directly retrieved based on a data topological structure. According to the method, on the premise that the system architecture is not changed, the distributed retrieval system can be quickly recovered when the fault occurs, the unavailable time of the system is shortened, and the availability, stability and resource utilization rate of the system are improved.

The invention provides a distributed system load balancing method and device, electronic equipment and a medium. The load balancing method for the distributed system is applied to a scheduling node in the distributed system, the distributed system comprises the scheduling node and a plurality of computing storage nodes, indexes in the distributed system are decomposed into pieces and stored in the computing storage nodes, and the load balancing method for the distributed system comprises the steps that in response to an index change request, the index change request is sent to the computing storage nodes; performing an index change on the plurality of compute storage nodes; determining a first offset, a second offset and a third offset of each calculation storage node after the index is changed; and based on the first offset, the second offset and the third offset, migration of the fragments on the plurality of calculation storage nodes is carried out, so that the first offset on each calculation storage node after migration is smaller than a first threshold value, the second offset is smaller than a second threshold value, and the third offset is smaller than a third threshold value. The embodiment of the invention can be applied to various scenes such as a distributed retrieval analysis engine.

This application discloses a system optimization device, method, equipment, and medium based on a distributed retrieval system, relating to the field of computer technology. The device includes: a skew balancing strategy module for setting monitoring rules, monitoring scope, and monitoring index thresholds for the current distributed retrieval cluster; a dynamic balancing monitoring module for monitoring the skewness of the distributed retrieval cluster according to the monitoring rules, monitoring scope, and monitoring index thresholds; and a skew balancing management module for dynamically balancing and partitioning / layering management of the resource distribution and request distribution of the distributed retrieval cluster based on cluster monitoring information. This application achieves balanced optimization of the current distributed retrieval cluster by adding a skew monitoring mechanism, a skew balancing management mechanism, and a request partitioning / layering control mechanism to the sharded storage and retrieval process of the distributed retrieval system, maintaining a balance between resources and requests, thereby avoiding request anomalies and resource waste caused by skewness.

The invention provides a distributed medical image filing and desensitization system, and the system comprises a receiving and analyzing module which is used for receiving a request, analyzing and extracting a UID set, and verifying the legality of the request; the task fragmentation module is used for converting the UID set into an RDD and dividing the RDD into a plurality of partitions; the data pulling module is used for pulling a DICOM data set corresponding to the UID contained in the corresponding partition according to a preset process; the dynamic desensitization module is used for extracting metadata fields containing sensitive personal identifiers from the DICOM data set and executing corresponding desensitization strategies according to field attributes; the object storage module is used for regenerating a data stream for the desensitized DICOM data set, naming the data stream and forcibly deleting an old object; and the task monitoring module is used for recording the time consumption and the state of each UID task in the modules, writing the time consumption and the state into a distributed retrieval engine and carrying out visualization. According to the method, a PACS server component based on dcm4che3 is combined with a Spark distributed computing framework, so that distributed scheduling, desensitization and archiving processing of image archiving requests are realized.

The invention relates to the technical field of distributed storage information retrieval, in particular to a multi-source heterogeneous data retrieval method and system for marine environmental protection, and the method comprises the steps: constructing a marine data ontology library, and determining a cross-modal feature vector mapping space through a double-flow neural network; space-time reference unification is executed, and a standardized retrieval unit is packaged; enhancing the optical image based on the underwater physical model and generating a multi-modal joint index key; constructing a hybrid index structure to manage the three-dimensional spatio-temporal data; utilizing Bloom filter fingerprints and binary exclusive-OR logic to execute sea-land increment index synchronization; and generating an extended query vector based on the marine knowledge graph, and outputting a result in combination with cross-modal similarity matching and space-time constraint. According to the method, the technical defects of marine environment data modal isomerism, semantic association deficiency, non-uniform three-dimensional distribution and synchronous lag in a narrow bandwidth environment are overcome by constructing a deep fusion distributed retrieval architecture, and the retrieval precision ratio and the sea-land data collaboration efficiency are improved.

The present application provides a distributed big data storage system and method. The method determines the spatiotemporal polygon retrieval granularity corresponding to the distributed storage node based on the historical economic data in the distributed storage node; performs polygon retrieval judgment on the historical economic data in the distributed storage node according to the spatiotemporal polygon retrieval granularity to obtain a spatiotemporally associated historical data set, and obtains the spatiotemporal storage dimension characteristics corresponding to each historical economic data in the spatiotemporally associated historical data set; determines the storage location of the economic collection data based on the spatiotemporal storage dimension characteristics corresponding to each historical economic data, and uses the spatiotemporal storage dimension characteristics of the economic collection data as spatiotemporal retrieval information for distributed retrieval. The method can store and retrieve the data based on the spatiotemporal storage dimension characteristics of the data to be stored, thereby improving the retrieval efficiency of spatiotemporal polygon retrieval of the distributed storage node.

This invention relates to the field of internet search services and discloses a big data-based investment lead analysis system, comprising: a query parsing unit that calls an event sequence template defining meta-event types and standard relative time window constraint intervals; a distributed retrieval and aggregation unit that retrieves and aggregates meta-event types to entity objects; a temporal logic verification unit that statistically analyzes the global document frequency of meta-events and generates a scaling factor through reverse mapping, dynamically corrects the time window, and determines only entity objects that pass the corrected dynamic verification interval as qualified; and an object sorting unit that generates a sorting score based on the template matching completeness. This invention constructs a reverse coupling mechanism between density and window to adaptively compress temporal logic boundaries in high background noise environments, maintaining the retrieval signal-to-noise ratio and logical confidence.

The invention relates to the technical field of data processing, in particular to a semantic enhanced storage and distributed retrieval system for unstructured data, which collects target unstructured data through a data collection module, encodes the target unstructured data through a semantic analysis module, and stores the encoded target unstructured data into a database. The method comprises the following steps: acquiring target unstructured data, performing fuzzy data monitoring through a semantic enhancement module, performing data fragmentation storage through a semantic storage module, and acquiring a distributed parallel retrieval scheme through a distributed retrieval module. And semantic analysis, semantic enhancement and data storage are performed according to the target unstructured data, and a distributed parallel retrieval scheme is acquired in real time, so that the target unstructured data can be efficiently stored and retrieved while the semantics of the target unstructured data is enhanced.

The invention belongs to the technical field of data analysis, and discloses an engineering file authenticity evaluation method and system based on AI. The method comprises the following steps: a cloud data center constructs a project file conversion model, a authenticity evaluation model and an evaluation report generation model, deploys a block chain network, and performs distributed storage on all project files; performing distributed retrieval and tampering verification in the block chain network, extracting a target project file, and performing graph structure conversion on the target project file by using the project file conversion model to obtain real-time graph structure data; and performing authenticity evaluation on the real-time graph structure data by using an authenticity evaluation model, and performing evaluation report generation on a real-time authenticity evaluation result by using an evaluation report generation model to obtain a real-time authenticity evaluation report. According to the method, the problems of low security, incomplete verification mechanism, low AI model intelligence degree, low efficiency and poor accuracy in the prior art are solved.

According to the high-resolution remote sensing image cloud computing optimization distributed retrieval method, on the basis of a self-defined MapReduce architecture IO method, a Hadoop-based high-resolution remote sensing image distributed retrieval architecture with offline library building and online retrieval is realized; a target detection method is integrated into a retrieval architecture, a method and parameters of an optimal alternative region extraction effect are determined through comparison and analysis of multiple homogeneous region extraction methods and different parameter settings and calculation of time consumption and result precision, and target positioning during retrieval is realized; the method comprises the following steps of: customizing a remote sensing image reading format and an image partitioning mode by adopting a GDAL, extracting various features of an original remote sensing image in a Map stage, searching an optimal feature combination, generating a visual word bag model through a clustering algorithm based on MapReduce, constructing a high-resolution remote sensing image database and a feature database based on Hadoop, carrying out distributed storage on image data and feature data, and storing the image data and the feature data. Retrieval acceleration is realized through parallel computing; the high-resolution remote sensing image retrieval speed is high, and the precision is high.

The invention relates to a distributed retrieval enhancement generation framework construction method oriented to cross-domain data, and belongs to the field of data retrieval. According to the method, a distributed retrieval enhancement generation framework Multi-RAG is constructed, the Multi-RAG adopts a layered decoupling distributed architecture design, a centralized retrieval enhancement generation system is reconstructed into three mutually cooperative functional layers, namely a query routing layer, a distributed data node layer and a cooperative generation service layer, and each functional layer internally comprises a plurality of specialized modules. Core functions of a specific level are realized through organic cooperation among the modules. The Multi-RAG constructed by the method disclosed by the invention effectively solves the problem of cross-domain scattered data retrieval; and a cross-node information fusion mechanism based on global reordering is designed, so that retrieval results from different data fields can be effectively integrated.

The invention provides a high-availability data management system and method for a satellite constellation. The high-availability data management system comprises a data management portal module used for performing integrated display on archived data; the efficient data query and retrieval module is used for performing query and retrieval on the archived data by adopting a distributed retrieval service and taking a to-be-processed retrieval condition configured by a product model as supplement; the data archiving access and extraction module is used for performing data archiving on the data according to the data type, and performing near-line storage extraction or off-line extraction on the archived data after query and retrieval; the data management statistical analysis module is used for performing data statistics on the archived data according to the data type and displaying a data statistics result in a specified form; and the data management monitoring module is used for monitoring the running state of the system and starting an early warning mechanism when an abnormality is monitored. According to the method, the management effect on the high-availability data of the satellite constellation can be remarkably improved, and meanwhile, the user requirements can be better met.

The application discloses a kind of equipment full life cycle knowledge distributed retrieval method, it is related to knowledge retrieval technical field, this method uses HashMap and B+ tree to realize inverted index, when equipment full life cycle knowledge distributed retrieval, client only needs to send request to a node, avoid multiple node communication, each keyword is only stored in the machine corresponding to a node, without merging cross node data, and communication volume is greatly reduced.Simultaneously, under the condition that communication is unstable, fast retrieval can be realized, and it is suitable for efficient retrieval under the condition that communication cost between distributed systems is higher.

The invention provides a rare disease data retrieval method and system, and belongs to the technical field of data retrieval, and the method comprises the steps of cloud data center infrastructure construction, real-time data processing and feature generation, block chain distributed storage, retrieval information knowledge mapping, distributed retrieval and data extraction, and result visual display. By constructing a block chain database containing a PBFT consensus mechanism, secure storage and tampering prevention of rare disease data are realized, and data integrity and reliability are guaranteed; a multi-algorithm fusion retrieval engine is adopted, and standardized integration of heterogeneous data, dynamic updating of a knowledge graph and accurate generation of a keyword graph and a retrieval label are achieved; through a two-stage retrieval mechanism of knowledge mapping, coarse retrieval and fine retrieval, the retrieval accuracy and efficiency are improved; and in combination with a visual display module, a retrieval result is visually presented, the efficient acquisition requirement of clinical diagnosis and treatment and scientific research on rare disease data is met, a data island is broken, and cross-domain resource sharing is promoted.

The application discloses a communication data intelligent retrieval method and platform based on semantic analysis, and relates to the technical field of natural language processing.The method comprises the following steps: after receiving a user's original query text, binding a correlation retrieval permission policy based on a user ID, loading a user portrait to complete semantics to obtain a plurality of semantic equivalent completion sentences, extracting feature words and combining a domain knowledge graph to construct a plurality of hierarchical retrieval feature trees, then matching a distributed retrieval unit, dynamically superimposing retrieval to obtain a plurality of refined result sets, arranging and integrating the result sets in descending order according to metadata time stamps, and outputting a communication data time series retrieval sequence.The application solves the technical problem that semantic understanding is single in traditional communication data retrieval, which leads to low retrieval efficiency and difficulty in accurately positioning communication data, and achieves the technical effects of accurate semantic understanding of communication data, ordered and efficient retrieval, and accurate positioning of communication data meeting the user's intention.