315 results about "Metadata management" patented technology

The invention provides an engineering drawing intelligent processing system based on a ComfyUI, and the system comprises an algorithm service module which integrates a plurality of drawing processing algorithm assemblies, such as drawing layout analysis, OCR recognition, drawing direction detection, semantic understanding, drawing frame disassembly, and axis net recognition, and provides services for the outside through a standardized interface; the workflow arrangement module is used for packaging the algorithm components into parameterized drawing processing nodes based on a visual node configuration framework of a ComfyUI, and supporting flexible combination, real-time preview and batch execution of drawing processing flows; and the workflow management module is used for importing, analyzing and publishing the workflow configuration file in the ComfyUI format, and realizing metadata management, publishing verification, API document generation, interface registration and call authority control of the workflow. The method supports multi-type input of PDF and CAD drawings, is suitable for engineering drawing processing tasks such as drawing label extraction, sub-drawing recognition, drawing frame segmentation and drawing filing, and is high in intelligent degree, flexible in process configuration, easy to integrate and high in expansibility.

The invention provides a beach big data geographic space intelligent application service method based on a large model Agent, and relates to the technical field of geographic space data artificial intelligence processing and analysis. According to the method, a distributed platform comprising a parallel computing cluster, a spatial database and a large language model is constructed, and standardized storage and metadata management of multi-source remote sensing, vector and attribute data are supported. The system generates a spatio-temporal query instruction through natural language analysis, completes cross-source matching and spatial calculation, outputs an analysis result in a multi-modal form such as a map, a chart and a text, and realizes intelligent and interactive service of geographic data processing.

The embodiment of the invention provides a multi-modal low-altitude grid data dynamic indexing method and device for improving a cache mechanism. The method comprises the following steps: mapping multi-modal low-altitude grid data to a unified vector space, and constructing a cross-modal semantic association hybrid index for the mapped low-altitude grid data; dividing the mapped low-altitude grid data into a plurality of dynamic sub-regions according to data distribution density and data access popularity, and managing a mixed index corresponding to each dynamic sub-region by combining a Loki log index fragmentation mechanism and tagged metadata; the mapped low-altitude grid data is cached by adopting an EHCD hierarchical multi-level cache architecture; and carrying out cross-modal low-altitude grid data retrieval by combining SIMD instruction set parallelization vector similarity calculation and GPU acceleration complex space relation operation. In this way, the real-time retrieval efficiency, the system expansibility and the energy efficiency ratio of the low-altitude grid data can be remarkably improved.

The invention discloses a metadata prefetching method and device, a storage medium, electronic equipment and a program, and relates to the field of metadata management of a distributed storage system.The metadata prefetching method comprises the steps that a first information block corresponding to a metadata operation event in a target storage area of each node is obtained; updating the three-dimensional correlation degree between any two metadata in each node according to the first information group; determining a prefetching priority corresponding to any two pieces of metadata through the updated three-dimensional correlation degree; and generating a heat association list of the metadata in each node according to the prefetching priority, and respectively sending the heat association lists corresponding to the plurality of nodes to the client so as to indicate the client to prefetch the target metadata in the plurality of nodes according to the plurality of heat association lists. Through the method and the device, the problem of low pre-fetching accuracy caused by dependence on static path association of the metadata and poor cross-node adaptability of a pre-fetching method for the metadata in a distributed storage system is solved, and the pre-fetching accuracy is improved.

The invention relates to an event trigger system based on an openGauss database. The system comprises: an event trigger definition module supporting a user to define a trigger rule of an event trigger; the trigger function execution module provides an execution environment for a function associated with the trigger and executes function logic; and the event monitoring and processing module monitors database operation events, matches corresponding event triggers according to trigger rules, and coordinates the execution sequence and error processing of correlation functions of the event triggers. According to the system, flexible trigger rule definition is supported through extension grammar, the metadata management efficiency is guaranteed by combining semantic verification and a special storage structure, the execution performance is optimized by utilizing pre-compiling and caching technologies, accurate monitoring and trigger coordination of events are realized through a kernel hook mechanism, and the reliability of the system is improved. The limitation that a traditional trigger only covers a DML event is effectively solved, the reliable management and control capability of full-type operation (DML / DDL) is provided for the openGauss database, and the safety, maintainability and application stability of the database are remarkably improved.

The invention provides an interpretable building contract risk review method and system with a mechanism layer knowledge graph, and the method comprises the steps: receiving a registered contract basis and a to-be-reviewed contract, and obtaining a standardized clause unit set; obtaining an executable mechanism library based on a large language model LLM; inputting, analyzing and normalizing values in a contract to be examined according to a mechanism, compiling and executing a mechanism formula, and obtaining a mechanism output result and a risk node aggregation probability; constructing a structured evidence packet and driving the knowledge graph to generate a risk level, a reason chain and a mending suggestion; and assembling the clause unit, the mechanism node and the risk node into a knowledge graph for exporting, and generating an interpretation path score for review and LLM reference. According to the method, the mechanism layer is introduced into the term layer and the risk layer of the knowledge graph, a causal chain between term parameters and risk triggering logic is explicitly represented, meanwhile, mechanism nodes are managed through formula and versioning metadata, and when market parameters change, an overall model does not need to be retrained or a rule base does not need to be modified on a large scale.

The invention belongs to the technical field of computer server management, and particularly discloses a laboratory collaborative environment-oriented user and data full-life-cycle server management system, which comprises a container virtualization layer, a server management layer and a server management layer, dynamic resource allocation is supported through a state saving-resource reallocation-environment recovery mechanism; the heterogeneous transaction coordination layer adopts a Saga transaction mode to coordinate a heterogeneous system so as to realize cross-system transaction consistency; the data collaboration ecological layer establishes typed management of a personal data volume, a shared software environment volume and a public data set volume, and supports software environment encapsulation sharing and versioning management; and the metadata management layer realizes real-time update and high-performance query of metadata based on a Redis hot update architecture. According to the method, the problems of inflexible resource allocation, incomplete environment isolation, dilemma in authority management and difficulty in data collaboration and recovery in the prior art are solved.

The invention relates to the technical field of big data storage, and discloses a multi-level data storage and mixed query method and system, a storage medium and equipment, and the method comprises the steps: collecting original buried point data, carrying out real-time stream processing, storing the processed real-time data in a hot storage layer, carrying out timing offline batch processing on historical buried point data, and storing the processed real-time data in a hot storage layer; converting into an optimized storage format and storing in an object storage system; an extended metadata management system is constructed, and structure information, partition information and access frequency information of data in the object storage system are automatically extracted and managed; executing query, and reducing the data scanning amount through a query optimization strategy; automatically migrating data among different storage layers through a dynamic cold and hot data layering mechanism; a unified query interface is provided, a query request is distributed to a corresponding real-time processing system or an offline query system based on an intelligent routing strategy, and a query result is cached and returned, so that high-performance and low-cost storage and second-level query of data are realized.

The invention discloses a metadata management method based on ES cold and hot data read-write separation, and relates to the technical field of metadata management, comprising: marking an SSD disk as a hot node, and marking an SATA disk as a cold node; assigning label parameters in a node starting command to enable the nodes to join according to roles; creating an index template to divide a hot index and a cold index, writing a default route to a hot node, and reading an automatic route of a coordination node; setting an automatic index rolling strategy, writing an active index into a hot node, automatically establishing a new index when a threshold value is reached, and preferentially allocating resources to the hot index; an ILM migration rule is configured, and the overdue index is automatically migrated to a cold node; writing real-time metadata into a hot node, migrating overdue metadata to a cold node regularly or manually, and deleting cold node data according to a strategy; real-time data are read from the hot nodes, historical data are read from the cold nodes, and on-demand access is achieved. The performance, reliability and resource utilization rate of the ES cluster are remarkably improved on the premise that the cost is controllable.

The invention discloses a distributed block storage method and device, electronic equipment and a storage medium. The load index is calculated by collecting the multi-dimensional performance indexes of the storage nodes in real time, and the data fragment storage position is dynamically adjusted according to the load. The self-adaptive multi-dimensional load sensing model can accurately adapt to various service scenes, real-time balance of loads between nodes is achieved, hot nodes are effectively avoided, and the overall performance of the system is improved. And when the load index exceeds the preset threshold value, hot data identification, node affinity evaluation and migration income calculation modes are started to perform data fragmentation migration, so that the utilization rate of mixed medium resources is improved, a high-performance storage medium is reasonably utilized, and the data access efficiency is improved while the storage cost is reduced. Through cooperative work of load monitoring, fragmentation decision making, migration execution, metadata management and the like, a fragmentation strategy is automatically optimized according to input and output characteristics of different service types, and the overall performance under different service scenes is improved.

The invention discloses a Text2SQL (Structured Query Language) system and method based on cue words. The system comprises a natural language input module for receiving and preprocessing a natural language query statement; the metadata management module is used for managing database metadata information; the cue word generation module is used for generating cue words containing database modes, user problems and SQL requirements on the basis of the query statements and the metadata; the large language model module is used for generating a preliminary SQL based on the cue word; the SQL grammar verification module is used for verifying grammar and feeding back errors; the manual auditing module is used for auditing and editing the SQL statements passing the verification; and the SQL execution module is used for executing the SQL passing the audit and returning a result. According to the method, the accuracy is improved by combining cue word guidance and a large model, the field adaptability is enhanced by relying on metadata, the data security is ensured by introducing manual auditing, the accuracy and security of SQL generation in a complex query scene are remarkably improved, and the development and maintenance cost of the system is reduced.

The invention discloses a big data governance method and system based on a rule engine, belongs to the technical field of big data processing, and aims to solve the technical problems of low efficiency, slow response and rule stiffness in the current data governance process. Comprising a data access layer used for collecting data from each data source; the metadata management module is used for constructing a knowledge graph based on the metadata; the rule engine module is used for constructing a rule base; the data quality evaluation module is used for analyzing and evaluating the multi-source heterogeneous data based on the rule base; the authority control and compliance auditing module is used for performing authority control and performing compliance analysis on data operation through a preset compliance scale board based on an auditing log; the governance engine module is used for carrying out governance operation on the multi-source heterogeneous data based on the data quality analysis result to form a data governance execution result; and the feedback optimization module is used for optimizing the AI large model based on the data governance execution result.

The invention discloses a distributed file management system and method, and belongs to the technical field of computer software, the system adopts a tree hierarchical structure to organize files, and the system comprises an application layer used for receiving file operation requests from a plurality of independent clients, and each request carries a unique application identifier; the file system layer comprises a configuration module, an operation module, a metadata management module, a day module, a timed task module and a gateway module, and the request is routed to a corresponding logic storage partition in the data layer according to an application identifier; the data layer comprises a database cluster and an object storage cluster, and performs persistent storage and management of logic isolation on metadata and file data according to application identifiers; and the infrastructure layer provides an extensible server and network resources. Based on a distributed architecture, load balancing and efficient access are realized through data storage partitioning, a metadata management and index technology and a client load balancing technology, and the expandability, fault tolerance and performance of the system are improved.

The invention provides a storage device, a storage method and a storage access method based on lake and warehouse integration. The storage device comprises a storage management unit, a unified metadata management unit and a federal query engine unit, the storage management unit is used for realizing hybrid storage and dynamic hierarchical storage; the unified metadata management unit is used for metadata atlas construction, concurrency control and GDPR compliance support; the federal query engine is used for intelligent routing decision, SQL dialect unification and intelligent push-down optimization; based on a lake and warehouse integrated storage device and an access technology thereof, unified management of structured and unstructured data is supported, the problem of data islands is avoided, and cross-engine data visibility is realized through unified metadata management; according to the lake and warehouse integrated storage device and the access method, technologies such as a dynamic hierarchical storage strategy, intelligent routing decision, intelligent push-down optimization and the like are fused, and the storage and access efficiency and the robustness of the system are greatly improved.

The invention relates to the field of distributed storage, and discloses a distributed memory system based on erasure codes and a dynamic repair mechanism. Comprising a data node cluster, a metadata management cluster, a bandwidth token management module, a dynamic repair composer and a repair agent node. And the data node cluster divides the object into strip generation data fragments and verification fragments and dispersedly stores the data fragments and the verification fragments according to a placement rule. And the metadata management cluster maintains object stripe mapping, a fragment position set, erasure code protection configuration and a version epoch, and records a repair intention log. And the bandwidth token management module grants repair execution permission according to the fault domain budget. The dynamic repair composer generates availability repair and durability repair tasks, the temporary check fragments are written in when the number of the available check fragments is lower than a threshold value in the availability stage, and the configuration fragments are recovered and the temporary fragments are recovered in the durability stage. And the consistency of version epochs is verified before the agent node is repaired and written back, and if not, recalculation is carried out.

The invention discloses a distributed medical image retrieval and hierarchical storage management system and method. The system comprises an interface processing module, a distributed image storage cluster, a routing forwarding module, a metadata management module and a storage management module. The interface processing module receives the image retrieval request, generates a routing identifier for positioning the storage position of a target medical image file and sends the routing identifier to the routing forwarding module; and the routing forwarding module queries a mapping relationship between the medical image file and the storage node and / or the storage hierarchy based on the routing identifier, determines a first storage node and forwards the request, so that the first storage node returns a target medical image file. And the storage management module obtains the access statistical information, determines a target storage level and / or a target storage node, migrates the medical image file, and triggers updating of the mapping relation after migration is completed. Therefore, cross-node accurate positioning and unified retrieval are realized, positioning consistency is kept after file migration, and hierarchical storage management based on access conditions is supported.

The invention discloses a data knitting cross-library view efficient query method and system, and relates to the technical field of data processing. The system comprises a unified query interface module, a metadata management module, an intelligent distributed query optimizer, a multi-level cache module, a materialized view management module, an execution engine module and a performance monitoring and feedback module. According to the efficient query method and system for the data knitting cross-library view, a dynamic cost model is constructed through an intelligent distributed query optimizer, and a push-down strategy is decided and queried in combination with source capability metadata; a multi-level cache and materialized view collaboration mechanism is adopted to reduce source data access; maintenance overhead of materialized views is reduced with incremental refresh driven by CDC.

The invention belongs to the field of artificial intelligence, particularly relates to an integrated artificial intelligence analysis and modeling platform supporting large-scale data processing, and aims to solve the problems of low resource scheduling efficiency, model training reasoning splitting, data algorithm coupling, limited cross-modal modeling and the like under high-dimensional heterogeneous mass data. The platform comprises a distributed data access layer, a unified metadata governance engine, a heterogeneous computing resource scheduling center, a model life cycle management module, a multi-modal fusion modeling framework and a dynamic service orchestration interface, and realizes automatic data access governance, dynamic resource scheduling, model full-cycle management and control and cross-modal intelligent fusion. And the resource utilization rate, the modeling accuracy and the service deployment efficiency are remarkably improved.

The invention relates to the technical field of power data processing, in particular to a dynamic cleaning and quality evaluation system for power multi-source heterogeneous data, which comprises a data access module for acquiring and preprocessing the power multi-source heterogeneous data. The metadata management module extracts metadata features and constructs a data quality evaluation index system, the generative adversarial network module comprises a generator and a discriminator, the generator repairs problem data, the discriminator evaluates the quality of the repaired data and feeds back the quality to the generator, and the rule engine module generates an adaptive cleaning rule based on the metadata features. The system is applied to the repairing process of the generator, the quality evaluation module evaluates the quality of the repaired data based on a multi-dimensional standard and generates an evaluation report, the data storage module stores the repaired data and the evaluation report, the system realizes integrated processing of data quality evaluation and repairing, the data processing efficiency is improved, and the data quality evaluation and repairing efficiency is improved. Compared with a traditional method, the efficiency is improved by 3-5 times, and an effective solution is provided for power multi-source heterogeneous data processing.

The invention relates to the technical field of computer data writing, in particular to a computer data writing system and method.The computer data writing system comprises a multi-level storage module, a data receiving and analyzing module, a heat evaluation and layering module, a differential writing module, a dynamic migration module, a metadata management module and a monitoring and scheduling module; according to the corresponding method, a hot, warm and cold three-level storage architecture is constructed, an initial data storage hierarchy is determined based on a hot degree prediction model, and differential write-in operation is executed; establishing a'migration forecast-write optimization 'cooperation mechanism, a'write performance data-storage hierarchy parameter' closed-loop optimization mechanism and a'multi-scene write strategy-hierarchy-migration 'parameter mapping system to ensure data access transparency; the cold data storage cost is reduced while the high write-in performance of the hot data is ensured, the problem that the storage performance and the cost are difficult to balance in a time sequence big data scene is solved, and the method is suitable for high-frequency time sequence data storage scenes such as power monitoring and financial transactions.

Disclosed is an improved approach to manage hierarchical metadata for a database system. A hierarchical metadata structure pertaining to a hierarchical object structure of a multitenant database architecture for a plurality of tenants may be maintained where the multitenant database architecture comprising a container database (CDB) that includes a pluggable database (PDB). A request for an access to a metadata object in the hierarchical metadata structure for one or more database objects in the container database may be received. In response to the request for the access to the metadata, access to at least a portion of the hierarchical metadata structure is provisioned.

The invention provides an optimization method and system for improving the file real-time access speed of an electric power embedded operating system, and belongs to the technical field of electric power embedded systems. The method comprises the following steps: establishing a dynamic preloading mechanism based on the operation state of the power equipment; the dynamic preloading mechanism triggers file preloading operation by monitoring the working condition change of the equipment; designing an improved cache replacement algorithm; the improved cache replacement algorithm is used for dynamically adjusting a cache strategy in combination with power file access characteristics; constructing a multi-dimensional file access scheduling system; the multi-dimensional file access scheduling system allocates system resources according to task priorities; executing a file access full-process acceleration strategy; the file access full-process acceleration strategy comprises metadata management, data prefetching and version control. The file access delay can be shortened, and the system resource utilization rate can be improved.

The invention provides a mixed storage and unified query layer data retrieval system based on an AI cloud desktop, which belongs to the technical field of AI intelligent application and comprises a unified query access layer, an SQL optimizer, a vectorization execution engine, a mixed storage engine and a metadata management center. Column storage, inverted index and row storage are organically combined, services are provided for the outside through a unified SQL query layer, and intelligent routing and vectorization computing technologies are introduced. The method is characterized in that an optimal storage engine is automatically selected for execution according to a query mode, the problems that a single inverted index architecture is low in efficiency and high in resource consumption under the scenes of ad-hoc analysis, high-concurrency point search and large-scale aggregation are solved, and better comprehensive query performance, lower data storage cost and higher system expansibility than traditional Elasticsearch are achieved.

The invention relates to an intelligent file management system based on RAG and metadata full-life-cycle management, which is characterized by comprising a front-end application layer, a back-end service layer, an RAG knowledge base analysis layer and a data storage layer, and all the layers work cooperatively through standardized interfaces; the RAG knowledge base analysis layer is constructed by taking a self-built RAG knowledge base as a core, and the back-end service layer encapsulates a metadata management center. According to the method, through a multi-dimensional retrieval mode of'metadata screening + semantic retrieval + knowledge graph association ', the retrieval range is narrowed by metadata, then a knowledge unit is accurately positioned through RAG semantic retrieval, and finally associated document information is supplemented, so that compared with traditional full-text retrieval, the retrieval precision is improved, and the retrieval response time is shortened to millisecond level; automatically mining a document association relationship through a knowledge graph construction module, and supporting a user-defined association rule; and through a document-knowledge integrated view and a scenarized knowledge aggregation function, cross-department and cross-project knowledge rapid circulation is realized, and the knowledge reuse efficiency is improved.

The invention relates to the technical field of intelligent fire stations, in particular to an intelligent modularized prefabricated cabin type tunnel fire station terminal dynamic early warning system based on edge calculation and multi-source perception. Comprising a multi-source data acquisition unit; the edge calculation processing unit is used for performing real-time fusion and dynamic modeling on the acquired data, fusing multi-source data based on Kalman filtering and LSTM, and constructing a risk evolution model with fast and slow dynamic components by adopting a singular perturbation theory; an intelligent early warning analysis unit; and a data management unit. According to the method, a multi-scale feature extraction framework is constructed by adopting parallel GRU branches through an improved long-short-term memory network: a short-term branch captures second-level rapid features such as flame flicker and sudden change of smoke concentration, and a long-term branch captures minute-level slow change trends such as temperature gradient rise; dynamic weights are given to different time scale features in combination with an adaptive attention mechanism; the design improves the pertinence and accuracy of feature extraction in a complex scene.

The invention discloses a distributed file system metadata full-memory management method and system.The distributed file system metadata full-memory management method comprises the steps that distributed file system metadata are stored in a memory and a storage layer through the same layout, and the same layout means that memory pages in the memory and storage blocks in the storage layer are the same in size; and taking the memory page in the memory as a page cache of a corresponding storage block in the storage layer, responding to a read request and a write request of a client through the distributed file system metadata stored in the memory, and directly writing the memory page written by the write request into the storage layer when the write request is received so as to realize persistent storage of the distributed file system metadata. The distributed file system metadata full-memory management method aims at achieving distributed file system metadata full-memory management, read operation completely gets rid of the I / O bottleneck, all storage bandwidths are dedicated to write operation, memory bandwidths are fully utilized for distributed file system metadata management, and persistence overhead is reduced.

The invention discloses an NUMA perceptible distributed parallel file system metadata management method and system.The method comprises the steps that a local metadata view and a global metadata view are constructed, the local metadata view is used for dynamically distributing metadata to nodes according to an access mode and maintaining independent metadata copies at the nodes, and the global metadata view is used for dynamically distributing the metadata to the nodes according to the access mode; the global metadata view is used for providing consistency and persistence support for cross-node operation of metadata; the method comprises the following steps: dynamically distributing metadata to each node according to an access mode by adopting a metadata partitioning strategy, and for high-frequency access directory sub-trees, finishing the creation of directories and files of the directory sub-trees at local nodes through a local priority rule; and for other metadata, the metadata under the same directory belong to the same node through dynamic distribution of a consistent Hash algorithm. According to the invention, the cross-node remote access overhead can be reduced, and the cross-node access efficiency is improved.

The invention provides a multi-source data integration method, a storage medium, a program product and electronic equipment, and can be applied to the field of financial science and technology. The method comprises the following steps: calling a corresponding adapter to read original data from each data source according to the types of a plurality of data sources; performing quality evaluation on the original data by adopting a random forest algorithm, determining to-be-cleaned data in the original data, and performing data cleaning on the to-be-cleaned data to obtain cleaned data; standardizing the cleaned data and uncleaned data in the original data by adopting a metadata management technology; under the condition that data conflicts exist in the standardized data, conflicting data are arbitrated so as to determine real data from the conflicting data, and the data conflicts represent that logic conflicts exist in data information from different data sources. The problem that a multi-source data integration scheme in the prior art is low in integration quality is solved.

The invention discloses a large model localization deployment and security isolation device for a hospital closed network environment, which consists of a model deployment unit, a security isolation unit, a data management unit, a security control unit and an intelligent updating unit, and realizes independent operation of a large model in a hospital. According to the system, safe input of medical data is achieved through data cleaning, desensitization and formatting processing, full-process safety of data and model files is guaranteed through authority control, access recording and encrypted storage, and credible upgrading and abnormal rollback of a model are completed through a one-way isolation and offline updating mechanism. The large model can be operated and maintained under the condition that an external network is not connected, the requirements of hospitals for data privacy and network security are met, and the method is suitable for various types of intelligent medical services.

The invention relates to a software and hardware combined fine-grained memory protection mechanism, which realizes high-speed mapping from a physical address to a fine-grained permission label by integrating a metadata search unit and a metadata conversion lookup buffer on a critical path of a processor loading / storage unit. An operating system maintains a multi-level fine-grained permission metadata table in a main memory, and the minimum memory protection granularity is refined to a 64-byte sub-page level. When a processor executes a memory access instruction, address conversion and permission verification are completed in parallel, an access type and a permission label are compared in real time within 1-2 clock periods, and when permission conflicts are detected, high-priority abnormity is triggered immediately, and illegal addresses and fault types are reported accurately. The method supports instruction set extension, buffer overflow protection, multi-level metadata management and user mode and kernel mode differentiated authority control, reduces the influence on the performance of the processor while improving the security of the memory, and is suitable for a computing system with high security and high performance.