458 results about "Metadata management" patented technology

The invention discloses a cross-system heterogeneous metadata intelligent acquisition method and system based on LLM, and the method comprises the following steps: S1, collecting and preprocessing heterogeneous system data, and constructing a heterogeneous graph; s2, analyzing nodes by the large language model, and splicing structural features to form initial representation of the nodes; s3, constructing a heterogeneous graph Transform, executing multi-layer propagation and low-rank decomposition, and outputting final node representation; s4, generating a field alignment path, an interface mapping path and a dependency path, and planning an acquisition path and an acquisition sequence; and S5, executing an acquisition task, converting a structured format, writing into a metadata center, and performing closed-loop optimization. According to the method, intelligent acquisition, semantic analysis and automatic integration of cross-system heterogeneous metadata are realized, and the efficiency and intelligent level of metadata management in a multi-source data environment are greatly improved.

The invention belongs to the technical field of data storage, and discloses a multi-source marine environment data storage method and system based on HDF5. According to the method, a hierarchical storage framework is constructed, a domain-type-source three-level directory architecture is constructed, a space-time dimension dynamic extension mechanism is executed, and space-time composite coding is performed; designing an HDF5 nested packet storage architecture, wherein grid topological structure definition, dynamic resolution self-adaptive storage, multi-source data processing and metadata management are carried out; constructing distributed data association and indexes, including data standardization preprocessing and cross-source data association; the storage performance is optimized and expanded, wherein density sensitive blocking and layered compression are carried out. According to the method, redundant storage of a high-latitude region is greatly reduced, and dynamic adaptation of a local encryption grid is supported. According to the invention, open circulation and collaborative utilization of interdisciplinary data resources are facilitated.

The invention relates to the technical field of cloud computing and distributed storage, and discloses a distributed metadata management method and storage system based on cloud computing, and the method comprises the following steps: S1, collecting the access frequency of a plurality of metadata objects in a set time period; s2, performing weighting processing on the access frequency, and constructing access heat distribution under multiple time scales; s3, obtaining the available cache capacity of a plurality of cache nodes, and constructing cache resource weight distribution; s4, assessing the access cost and migration cost between each metadata object and each cache node; and S5, generating a scheduling strategy between the metadata object and the cache node based on a preset optimization target, wherein the optimization target is access cost minimization. According to the method, a heat, resource and cost joint modeling mechanism is constructed, and optimal scheduling and feedback adaptive control are combined, so that accurate decision, dynamic adjustment and migration cost controllability of copy deployment are realized, and the performance and stability of distributed metadata management are effectively improved.

The invention provides a data knitting system and method based on dynamic metadata identification and DAG optimization, and relates to the technical field of data knitting, the data knitting system comprises a metadata management layer, a data processing layer and an execution layer, the metadata management layer comprises a dynamic metadata identification module and a labeling storage engine module, the data processing layer comprises an adaptive DAG optimization engine module and a dynamic task distributor module, and the execution layer comprises a data computing node cluster module and a multi-source data connector module; the method comprises the steps that a DAG task scheduling strategy is established, and dynamic optimization is achieved; according to the method, the service labels and DAG task scheduling are deeply bound to form a'data-calculation-service 'closed loop, the execution sequence and resource allocation of the tasks can be adjusted according to the service labels changing in real time, so that the task scheduling not only considers the use of calculation resources, but also can optimize the tasks in real time according to service requirements, and the task scheduling efficiency is improved. Therefore, the overall treatment efficiency is improved.

The invention provides an engineering drawing intelligent processing system based on a ComfyUI, and the system comprises an algorithm service module which integrates a plurality of drawing processing algorithm assemblies, such as drawing layout analysis, OCR recognition, drawing direction detection, semantic understanding, drawing frame disassembly, and axis net recognition, and provides services for the outside through a standardized interface; the workflow arrangement module is used for packaging the algorithm components into parameterized drawing processing nodes based on a visual node configuration framework of a ComfyUI, and supporting flexible combination, real-time preview and batch execution of drawing processing flows; and the workflow management module is used for importing, analyzing and publishing the workflow configuration file in the ComfyUI format, and realizing metadata management, publishing verification, API document generation, interface registration and call authority control of the workflow. The method supports multi-type input of PDF and CAD drawings, is suitable for engineering drawing processing tasks such as drawing label extraction, sub-drawing recognition, drawing frame segmentation and drawing filing, and is high in intelligent degree, flexible in process configuration, easy to integrate and high in expansibility.

The invention discloses an agricultural crop monitoring system and method based on multi-source remote sensing data, and relates to the technical field of agricultural crop production, and the system comprises a ground remote sensing platform which is used for regularly collecting crop growth data and carrying out the recognition and prediction operation of crop growth vigor, the ground remote sensing platform comprises a crop region division unit, a data analysis unit, a data management server, a memory and a display panel; the monitoring device is used for carrying out omnibearing monitoring on the growth condition of agricultural crops, the ground remote sensing platform and the monitoring device communicate with each other through a network, and the monitoring device comprises a ground monitoring station and unmanned aerial vehicle equipment. Compared with a traditional manual monitoring mode and a single remote sensing monitoring mode, human resource cost can be reduced to a certain extent, crop growth conditions can be more comprehensively known through the unmanned aerial vehicle, the ground monitoring station and the like, the monitoring range is increased, the monitoring precision is improved, and abnormal changes in the crop growth process are found in time.

A system (100) for adaptive data blending across heterogeneous BI platforms for real-time decision making, comprising: (a) a data source abstraction module configured to establish standardised and secure connections with several different Business Intelligence (BI) platforms and convert their native data structures into a unified intermediate schema; (b) a real-time data ingestion and synchronisation module operable to continuously monitor and retrieve data updates from the BI platforms using event-driven mechanisms and configurable synchronisation policies; c) an adaptive data mapping and transformation module designed to intelligently harmonize, adapt, and semantically translate disparate data formats using rule-based logic and AI-assisted schema mapping; (d) a semantic context and metadata management module configured to maintain a unified data dictionary, track data sequence and maintain semantic consistency across platforms; (e) a cross-platform query orchestration engine designed to decompose common user queries, translate them into platform-specific query languages, execute them across the relevant BI platforms, and aggregate the results into a coherent output; (f) a decision intelligence and recommendation module that analyses the mixed data to generate real-time insights, alerts and contextual recommendations for decision support; and (g) a security, governance and compliance module configured to enforce access controls, maintain audit trails and ensure compliance with data protection regulations; h) the system provides seamless, real-time and secure data blending to enable consistent analytics and decision-making across heterogeneous BI environments.

A data pipeline validation system and method configured to partially automate testing of data pipelines in a distributed computing environment. The system includes a data pipeline analytic device equipped with various modules, such as a query generation module, data frame comparison module, and metadata management module. The query generation module employs natural language processing techniques to analyze configuration entries and dynamically generate SQL queries tailored to specific test cases. The data frame comparison module compares the results of different test cases using distributed collections, enabling parallel processing and efficient result comparison. The metadata management module captures and stores relevant metadata for traceability and auditing purposes. The system facilitates comprehensive validation of data pipelines, enabling organizations to ensure the accuracy, reliability, and integrity of data.

The invention provides a beach big data geographic space intelligent application service method based on a large model Agent, and relates to the technical field of geographic space data artificial intelligence processing and analysis. According to the method, a distributed platform comprising a parallel computing cluster, a spatial database and a large language model is constructed, and standardized storage and metadata management of multi-source remote sensing, vector and attribute data are supported. The system generates a spatio-temporal query instruction through natural language analysis, completes cross-source matching and spatial calculation, outputs an analysis result in a multi-modal form such as a map, a chart and a text, and realizes intelligent and interactive service of geographic data processing.

The embodiment of the invention provides a multi-modal low-altitude grid data dynamic indexing method and device for improving a cache mechanism. The method comprises the following steps: mapping multi-modal low-altitude grid data to a unified vector space, and constructing a cross-modal semantic association hybrid index for the mapped low-altitude grid data; dividing the mapped low-altitude grid data into a plurality of dynamic sub-regions according to data distribution density and data access popularity, and managing a mixed index corresponding to each dynamic sub-region by combining a Loki log index fragmentation mechanism and tagged metadata; the mapped low-altitude grid data is cached by adopting an EHCD hierarchical multi-level cache architecture; and carrying out cross-modal low-altitude grid data retrieval by combining SIMD instruction set parallelization vector similarity calculation and GPU acceleration complex space relation operation. In this way, the real-time retrieval efficiency, the system expansibility and the energy efficiency ratio of the low-altitude grid data can be remarkably improved.

The invention discloses a metadata prefetching method and device, a storage medium, electronic equipment and a program, and relates to the field of metadata management of a distributed storage system.The metadata prefetching method comprises the steps that a first information block corresponding to a metadata operation event in a target storage area of each node is obtained; updating the three-dimensional correlation degree between any two metadata in each node according to the first information group; determining a prefetching priority corresponding to any two pieces of metadata through the updated three-dimensional correlation degree; and generating a heat association list of the metadata in each node according to the prefetching priority, and respectively sending the heat association lists corresponding to the plurality of nodes to the client so as to indicate the client to prefetch the target metadata in the plurality of nodes according to the plurality of heat association lists. Through the method and the device, the problem of low pre-fetching accuracy caused by dependence on static path association of the metadata and poor cross-node adaptability of a pre-fetching method for the metadata in a distributed storage system is solved, and the pre-fetching accuracy is improved.

A data deduplication method includes: receiving a write request with first data block; writing the first data block into a storage device; writing metadata of the first data block into a first partition that is in a plurality of partitions of a metadata management structure and that is determined based on a feature of the first data block; and deleting the metadata of the first data block in the first partition and deleting the first data block from the storage device based on address information of the first data block when a fingerprint that is the same as the fingerprint of the first data block exists in the first partition. The method can prevent infrequently updated data from being evicted because resources are occupied by frequently updated data, thereby improving a deduplication ratio.

The invention relates to an event trigger system based on an openGauss database. The system comprises: an event trigger definition module supporting a user to define a trigger rule of an event trigger; the trigger function execution module provides an execution environment for a function associated with the trigger and executes function logic; and the event monitoring and processing module monitors database operation events, matches corresponding event triggers according to trigger rules, and coordinates the execution sequence and error processing of correlation functions of the event triggers. According to the system, flexible trigger rule definition is supported through extension grammar, the metadata management efficiency is guaranteed by combining semantic verification and a special storage structure, the execution performance is optimized by utilizing pre-compiling and caching technologies, accurate monitoring and trigger coordination of events are realized through a kernel hook mechanism, and the reliability of the system is improved. The limitation that a traditional trigger only covers a DML event is effectively solved, the reliable management and control capability of full-type operation (DML / DDL) is provided for the openGauss database, and the safety, maintainability and application stability of the database are remarkably improved.

The invention discloses a metadata version control and consistency verification method and system, and belongs to the technical field of distributed system metadata management, the method comprises the steps of version tree construction, version query and backtracking, version merging and conflict processing, the version tree construction takes an initial timestamp associated with an initial version of metadata as a root node of the version tree; whenever the metadata is changed, a unique timestamp is generated for the changed metadata; carrying out associated storage on the generated timestamp and the changed metadata; and creating a new node according to the newly generated timestamp, establishing a father-son relationship between the new node and the node of the previous version to form a branch structure of the version tree, and arranging the nodes in the version tree according to the sequence of the timestamp. According to the method, the management efficiency of multi-version metadata can be remarkably improved, the data integrity and consistency are effectively guaranteed, and the method is suitable for cloud storage system metadata management, cross-platform metadata migration and other scenes.

The invention discloses a distributed file system metadata index optimization method based on remote direct memory access, which comprises directory metadata index based on path name hash: defining a plurality of layers of directories in a directory tree as key directories, and regarding the path as a character string. After calculation of the hash function, determining the storage position of the data in the memory of the metadata server of the distributed file system; and optimizing the memory position of the file metadata based on the popularity, namely placing the file metadata which is frequently accessed at a position adjacent to the key directory metadata to which the file metadata belongs. A metadata management module in a metadata server of the distributed file system regularly checks the popularity of the file and carries out cold and hot migration according to the situation; and client cache optimization oriented to key directory metadata: a metadata cache of a distributed file system client caches the metadata and directory entries of the key directory. Optimally assuming that the metadata and directory entries of the key directory in the cache are not invalid, and updating the cache according to the result of the next-level metadata search. According to the method, the remote reading performance of the metadata index of the distributed file system is optimized by fully utilizing the characteristic of remote direct memory access, the network transmission frequency is reduced, and the load of the metadata server is effectively reduced.

The invention discloses a financial digital intelligent management system, and relates to the technical field of computers, and the system comprises an atomic power unit ACU, a side effect metadata management module which is used for decomposing a financial process into ACU and defining the structured side effect metadata of the ACU, and the structured side effect metadata comprises an influence object, a type, a condition, a range, a preset risk level and the importance of an associated data object; the dynamic process arrangement engine module is used for generating candidate execution paths according to the process definition and the ACU side effect metadata, and selecting a final execution plan by taking the score minimization as an optimization target; and the execution control and risk relief module selectively performs influence domain sandbox rehearsal before actual execution of the ACU according to a risk assessment result, and executes conditional rollback or compensation when execution fails or unacceptable side effects are generated. According to the invention, through prospective side effect management and intelligent risk avoidance and control, the stability, reliability, auditing performance and combinable innovation capability of a financial automation process are significantly improved.

Solutions for integration and ongoing management of enterprise applications operating in a networked environment are disclosed. An AI transformer extracts, transforms, and manages integration point information from various sources, including documents, UMLs, and other system diagrams. Instead of relying on traditional documentation, the invention utilizes blockchain-based NFTs (non-fungible tokens) to store and manage integration point information. NFTs are created using smart contracts, which define the rules and conditions for the NFT. A metadata engine prepares metadata for generating the Integration Point NFTs. An NFT manager creates the integration point NFTs. A change analyzer determines the impact of changes to integration points on different applications and functionalities. A release compliance component is integrated with DevOps and is responsible for ensuring that integration point changes are incorporated into all impacted applications according to the release defined in the NFT.

The invention provides an interpretable building contract risk review method and system with a mechanism layer knowledge graph, and the method comprises the steps: receiving a registered contract basis and a to-be-reviewed contract, and obtaining a standardized clause unit set; obtaining an executable mechanism library based on a large language model LLM; inputting, analyzing and normalizing values in a contract to be examined according to a mechanism, compiling and executing a mechanism formula, and obtaining a mechanism output result and a risk node aggregation probability; constructing a structured evidence packet and driving the knowledge graph to generate a risk level, a reason chain and a mending suggestion; and assembling the clause unit, the mechanism node and the risk node into a knowledge graph for exporting, and generating an interpretation path score for review and LLM reference. According to the method, the mechanism layer is introduced into the term layer and the risk layer of the knowledge graph, a causal chain between term parameters and risk triggering logic is explicitly represented, meanwhile, mechanism nodes are managed through formula and versioning metadata, and when market parameters change, an overall model does not need to be retrained or a rule base does not need to be modified on a large scale.

The invention belongs to the technical field of computer server management, and particularly discloses a laboratory collaborative environment-oriented user and data full-life-cycle server management system, which comprises a container virtualization layer, a server management layer and a server management layer, dynamic resource allocation is supported through a state saving-resource reallocation-environment recovery mechanism; the heterogeneous transaction coordination layer adopts a Saga transaction mode to coordinate a heterogeneous system so as to realize cross-system transaction consistency; the data collaboration ecological layer establishes typed management of a personal data volume, a shared software environment volume and a public data set volume, and supports software environment encapsulation sharing and versioning management; and the metadata management layer realizes real-time update and high-performance query of metadata based on a Redis hot update architecture. According to the method, the problems of inflexible resource allocation, incomplete environment isolation, dilemma in authority management and difficulty in data collaboration and recovery in the prior art are solved.

The invention provides an unstructured data lake hierarchical storage and verification system and method, and relates to the technical field of data storage and verification. The system comprises an application layer, a data lake interface layer, a metadata management layer, an unstructured data storage layer and a verification service layer, the application layer realizes data access and operation by calling services provided by the data lake interface layer; the data lake interface layer provides a unified access interface to realize unified input and output of structured and unstructured data; the metadata management layer is used for storing metadata information of user files and supporting multi-dimensional retrieval; the unstructured data storage layer is used for storing user files transmitted through the data lake interface layer, supporting various storage media and storage schemes, and automatically allocating the storage media based on a preset rule; and the verification service layer provides a data verification service. According to the method, automatic scheduling, unified access and multi-dimensional verification management of cold and hot hierarchical storage in the unstructured data lake are realized.

The invention relates to the technical field of big data storage, and discloses a multi-level data storage and mixed query method and system, a storage medium and equipment, and the method comprises the steps: collecting original buried point data, carrying out real-time stream processing, storing the processed real-time data in a hot storage layer, carrying out timing offline batch processing on historical buried point data, and storing the processed real-time data in a hot storage layer; converting into an optimized storage format and storing in an object storage system; an extended metadata management system is constructed, and structure information, partition information and access frequency information of data in the object storage system are automatically extracted and managed; executing query, and reducing the data scanning amount through a query optimization strategy; automatically migrating data among different storage layers through a dynamic cold and hot data layering mechanism; a unified query interface is provided, a query request is distributed to a corresponding real-time processing system or an offline query system based on an intelligent routing strategy, and a query result is cached and returned, so that high-performance and low-cost storage and second-level query of data are realized.

The invention relates to the technical field of metadata management, and discloses a method for improving metadata storage and query efficiency, comprising: establishing a meta-model of each type of metadata, the meta-model comprising an object, an attribute and a relationship of the metadata; obtaining a heat value of each meta-model based on historical information of the meta-models, performing dynamic heat partitioning on the meta-models according to the heat values, and adjusting an index strategy, a storage strategy and a cache strategy according to the dynamic heat partitioning of the meta-models; the storage and query process is monitored in real time, index data monitored in real time in different dynamic popularity partitions are analyzed, and the meta-model is adjusted according to the analysis result. According to the method, the corresponding management mode can be dynamically adjusted according to the popularity state of the meta-model, so that the metadata storage and query efficiency is improved on the whole; meanwhile, real-time optimization of the meta-model is achieved by monitoring the storage and query processes in real time, and the metadata storage and query efficiency is improved.

The invention discloses a metadata management method based on ES cold and hot data read-write separation, and relates to the technical field of metadata management, comprising: marking an SSD disk as a hot node, and marking an SATA disk as a cold node; assigning label parameters in a node starting command to enable the nodes to join according to roles; creating an index template to divide a hot index and a cold index, writing a default route to a hot node, and reading an automatic route of a coordination node; setting an automatic index rolling strategy, writing an active index into a hot node, automatically establishing a new index when a threshold value is reached, and preferentially allocating resources to the hot index; an ILM migration rule is configured, and the overdue index is automatically migrated to a cold node; writing real-time metadata into a hot node, migrating overdue metadata to a cold node regularly or manually, and deleting cold node data according to a strategy; real-time data are read from the hot nodes, historical data are read from the cold nodes, and on-demand access is achieved. The performance, reliability and resource utilization rate of the ES cluster are remarkably improved on the premise that the cost is controllable.

The invention discloses a distributed block storage method and device, electronic equipment and a storage medium. The load index is calculated by collecting the multi-dimensional performance indexes of the storage nodes in real time, and the data fragment storage position is dynamically adjusted according to the load. The self-adaptive multi-dimensional load sensing model can accurately adapt to various service scenes, real-time balance of loads between nodes is achieved, hot nodes are effectively avoided, and the overall performance of the system is improved. And when the load index exceeds the preset threshold value, hot data identification, node affinity evaluation and migration income calculation modes are started to perform data fragmentation migration, so that the utilization rate of mixed medium resources is improved, a high-performance storage medium is reasonably utilized, and the data access efficiency is improved while the storage cost is reduced. Through cooperative work of load monitoring, fragmentation decision making, migration execution, metadata management and the like, a fragmentation strategy is automatically optimized according to input and output characteristics of different service types, and the overall performance under different service scenes is improved.

The invention discloses a Text2SQL (Structured Query Language) system and method based on cue words. The system comprises a natural language input module for receiving and preprocessing a natural language query statement; the metadata management module is used for managing database metadata information; the cue word generation module is used for generating cue words containing database modes, user problems and SQL requirements on the basis of the query statements and the metadata; the large language model module is used for generating a preliminary SQL based on the cue word; the SQL grammar verification module is used for verifying grammar and feeding back errors; the manual auditing module is used for auditing and editing the SQL statements passing the verification; and the SQL execution module is used for executing the SQL passing the audit and returning a result. According to the method, the accuracy is improved by combining cue word guidance and a large model, the field adaptability is enhanced by relying on metadata, the data security is ensured by introducing manual auditing, the accuracy and security of SQL generation in a complex query scene are remarkably improved, and the development and maintenance cost of the system is reduced.

The invention discloses a big data governance method and system based on a rule engine, belongs to the technical field of big data processing, and aims to solve the technical problems of low efficiency, slow response and rule stiffness in the current data governance process. Comprising a data access layer used for collecting data from each data source; the metadata management module is used for constructing a knowledge graph based on the metadata; the rule engine module is used for constructing a rule base; the data quality evaluation module is used for analyzing and evaluating the multi-source heterogeneous data based on the rule base; the authority control and compliance auditing module is used for performing authority control and performing compliance analysis on data operation through a preset compliance scale board based on an auditing log; the governance engine module is used for carrying out governance operation on the multi-source heterogeneous data based on the data quality analysis result to form a data governance execution result; and the feedback optimization module is used for optimizing the AI large model based on the data governance execution result.

The invention provides a memory database starting method, system and device and a medium, and belongs to the technical field of computers. The method comprises the following steps: detecting a current project scale and a performance index of an original server, and determining whether to trigger a memory processing mechanism; if a memory processing mechanism is triggered, selecting a memory server according to high-performance computing and expansibility requirements, accessing an original server by utilizing a hot plug technology, configuring a memory database and initializing a memory computing cluster; configuring a to-be-synchronized database table by utilizing a metadata management function, loading original data to a memory database through a full-amount synchronization and incremental updating mechanism, and starting real-time data synchronization; starting a memory computing cluster, automatically detecting a memory processing state on an original server when a service task is triggered, and routing a large quantity of data tasks meeting conditions to a memory server to execute parallel computing. According to the method, the memory can be dynamically started for data processing, the operation efficiency is improved, and the user experience is improved.

The invention discloses a metadata management method and device, electronic equipment and a storage medium, and relates to the technical field of data storage, and the method comprises the steps: receiving a metadata operation request sent by a client, determining a target logic partition based on a mapping relation between an identifier and the logic partition, and determining a target physical sub-book group through a mapping relationship between the logic partition and the physical sub-book group, and selecting a target storage node to process the metadata operation request based on the storage node state information in the target physical sub-book group. According to the method, the most suitable storage node can be intelligently determined to execute the metadata operation by introducing a mapping mechanism between the identifier and the logic partition as well as between the logic partition and the physical sub-book group and combining the state information of the storage node; the problems that in traditional metadata management, a single server becomes a performance bottleneck and data searching and maintaining efficiency is low are solved, and the technical effects of improving system performance, improving data processing efficiency and enhancing system reliability are achieved.

The invention provides a data asset management system and method based on a data asset catalogue, and the system comprises a data classification and structure management module which is used for dividing data into examples, examples and cases, and carrying out the standardized definition, index establishment and retrieval support of the storage forms of all kinds of data through a metadata management module; the data asset life cycle management module is used for recording and controlling the state of the data assets in each stage in the life cycle from development to commercialization in the process of converting the data into the data assets, and describing process information of various artificial products generated by the data assets in the life cycle and mutual relations of the process information through the metadata management module; and the data asset blood relationship management module is used for recording dependency relationships between data sources and data applications and between the data applications through a hierarchical management strategy and a data asset reference model between different management levels so as to form a traceable chain between the data assets.

The invention discloses a distributed file management system and method, and belongs to the technical field of computer software, the system adopts a tree hierarchical structure to organize files, and the system comprises an application layer used for receiving file operation requests from a plurality of independent clients, and each request carries a unique application identifier; the file system layer comprises a configuration module, an operation module, a metadata management module, a day module, a timed task module and a gateway module, and the request is routed to a corresponding logic storage partition in the data layer according to an application identifier; the data layer comprises a database cluster and an object storage cluster, and performs persistent storage and management of logic isolation on metadata and file data according to application identifiers; and the infrastructure layer provides an extensible server and network resources. Based on a distributed architecture, load balancing and efficient access are realized through data storage partitioning, a metadata management and index technology and a client load balancing technology, and the expandability, fault tolerance and performance of the system are improved.