606 results about "Metadata management" patented technology

The invention discloses a distributed storage method and system for data security, and belongs to the technical field of data storage security, the method comprises the following steps: firstly, fragmenting original data, and generating a unique identifier for each data fragment; for each fragment, an encryption algorithm and key credentials are adaptively selected, and the key credentials are stored in a key management system. By performing homomorphic encryption on the key voucher, an encryption handle is generated, and efficient positioning and retrieval of the key are realized. According to the method, a differentiated encryption strategy is adopted, a most suitable encryption scheme is selected according to fragment characteristics, and encrypted data fragments are dispersed and stored in a plurality of distributed storage nodes. A global mapping index is constructed in a metadata management server, and a corresponding relation between an original data global identifier and metadata of each data fragment is recorded, including information such as identifiers before and after encryption, a storage position, an encryption algorithm, a key handle and the like. According to the method, the data leakage risk is effectively reduced, and the safety and reliability of the storage system are improved.

The invention discloses a cross-system heterogeneous metadata intelligent acquisition method and system based on LLM, and the method comprises the following steps: S1, collecting and preprocessing heterogeneous system data, and constructing a heterogeneous graph; s2, analyzing nodes by the large language model, and splicing structural features to form initial representation of the nodes; s3, constructing a heterogeneous graph Transform, executing multi-layer propagation and low-rank decomposition, and outputting final node representation; s4, generating a field alignment path, an interface mapping path and a dependency path, and planning an acquisition path and an acquisition sequence; and S5, executing an acquisition task, converting a structured format, writing into a metadata center, and performing closed-loop optimization. According to the method, intelligent acquisition, semantic analysis and automatic integration of cross-system heterogeneous metadata are realized, and the efficiency and intelligent level of metadata management in a multi-source data environment are greatly improved.

The invention provides an integrated metadata management method and system based on data source expandability, and relates to the technical field of data sources, and the method comprises the steps: executing real-time detection on a plurality of heterogeneous data sources through a PF4J plug-in framework, collecting connection information, and establishing a dynamic connection pool; obtaining a table structure, a field attribute and an association relationship list, and assembling the table structure, the field attribute and the association relationship list into JSON format data; performing analysis and standardization processing on the data, and constructing an association degree calculation matrix; generating a standardized data structure description file; establishing a three-layer index directory, and detecting structure change and updating; and calculating a data table reference dependency relationship, and generating a DAG data flow path. According to the method, efficient integration and management of heterogeneous data sources are realized, and the metadata processing efficiency is improved.

The invention belongs to the technical field of data storage, and discloses a multi-source marine environment data storage method and system based on HDF5. According to the method, a hierarchical storage framework is constructed, a domain-type-source three-level directory architecture is constructed, a space-time dimension dynamic extension mechanism is executed, and space-time composite coding is performed; designing an HDF5 nested packet storage architecture, wherein grid topological structure definition, dynamic resolution self-adaptive storage, multi-source data processing and metadata management are carried out; constructing distributed data association and indexes, including data standardization preprocessing and cross-source data association; the storage performance is optimized and expanded, wherein density sensitive blocking and layered compression are carried out. According to the method, redundant storage of a high-latitude region is greatly reduced, and dynamic adaptation of a local encryption grid is supported. According to the invention, open circulation and collaborative utilization of interdisciplinary data resources are facilitated.

The invention relates to the technical field of cloud computing and distributed storage, and discloses a distributed metadata management method and storage system based on cloud computing, and the method comprises the following steps: S1, collecting the access frequency of a plurality of metadata objects in a set time period; s2, performing weighting processing on the access frequency, and constructing access heat distribution under multiple time scales; s3, obtaining the available cache capacity of a plurality of cache nodes, and constructing cache resource weight distribution; s4, assessing the access cost and migration cost between each metadata object and each cache node; and S5, generating a scheduling strategy between the metadata object and the cache node based on a preset optimization target, wherein the optimization target is access cost minimization. According to the method, a heat, resource and cost joint modeling mechanism is constructed, and optimal scheduling and feedback adaptive control are combined, so that accurate decision, dynamic adjustment and migration cost controllability of copy deployment are realized, and the performance and stability of distributed metadata management are effectively improved.

The invention provides a data knitting system and method based on dynamic metadata identification and DAG optimization, and relates to the technical field of data knitting, the data knitting system comprises a metadata management layer, a data processing layer and an execution layer, the metadata management layer comprises a dynamic metadata identification module and a labeling storage engine module, the data processing layer comprises an adaptive DAG optimization engine module and a dynamic task distributor module, and the execution layer comprises a data computing node cluster module and a multi-source data connector module; the method comprises the steps that a DAG task scheduling strategy is established, and dynamic optimization is achieved; according to the method, the service labels and DAG task scheduling are deeply bound to form a'data-calculation-service 'closed loop, the execution sequence and resource allocation of the tasks can be adjusted according to the service labels changing in real time, so that the task scheduling not only considers the use of calculation resources, but also can optimize the tasks in real time according to service requirements, and the task scheduling efficiency is improved. Therefore, the overall treatment efficiency is improved.

The invention relates to the technical field of data storage, and discloses a lossless compression algorithm-based data storage method, which comprises the following steps of: receiving to-be-stored data through a data interface, automatically identifying a data format by using a data analysis tool, and performing feature analysis to obtain the size, the type and the complexity of the data; through deep analysis of data features, the method can flexibly select a proper Huffman coding processing mode, and optimizes a compression process according to different data types and complexity. In addition, a context modeling optimization mechanism is introduced, the coding mode is dynamically adjusted, and the compression ratio and the storage efficiency are further improved; efficient storage and reliability verification of data are ensured through a block storage strategy in combination with metadata management; the problem that an existing compression algorithm is low in efficiency is solved, the integrity and reliability of data storage are improved, and the method is suitable for storage requirements of large-scale data and diversified data.

The invention provides an engineering drawing intelligent processing system based on a ComfyUI, and the system comprises an algorithm service module which integrates a plurality of drawing processing algorithm assemblies, such as drawing layout analysis, OCR recognition, drawing direction detection, semantic understanding, drawing frame disassembly, and axis net recognition, and provides services for the outside through a standardized interface; the workflow arrangement module is used for packaging the algorithm components into parameterized drawing processing nodes based on a visual node configuration framework of a ComfyUI, and supporting flexible combination, real-time preview and batch execution of drawing processing flows; and the workflow management module is used for importing, analyzing and publishing the workflow configuration file in the ComfyUI format, and realizing metadata management, publishing verification, API document generation, interface registration and call authority control of the workflow. The method supports multi-type input of PDF and CAD drawings, is suitable for engineering drawing processing tasks such as drawing label extraction, sub-drawing recognition, drawing frame segmentation and drawing filing, and is high in intelligent degree, flexible in process configuration, easy to integrate and high in expansibility.

The invention discloses a multi-source water environment data processing method and system based on intelligent Internet of Things, and relates to the technical field of data processing, and the method comprises the steps: carrying out the standardization processing of obtained multi-source heterogeneous water environment monitoring data, obtaining the standardized data, carrying out the multi-scale decomposition, and obtaining low-dimensional data; a distributed storage architecture is adopted for low-dimensional data, the data are stored on a plurality of nodes in a scattered mode, a data index and metadata management mechanism is established, and data retrieval and access are conducted; cleaning, aggregating and analyzing the low-dimensional data by adopting a stream data processing method, extracting key indexes and events, setting a water quality parameter threshold value and a change rate threshold value, constructing a water quality abnormity early warning rule in combination with a decision tree method, and judging whether water quality abnormity exists or not; and constructing a water quality prediction model through a support vector machine and a random forest machine learning method, and predicting a water quality trend. According to the invention, powerful technical support is provided for water environment monitoring and management.

A data pipeline validation system and method configured to partially automate testing of data pipelines in a distributed computing environment. The system includes a data pipeline analytic device equipped with various modules, such as a query generation module, data frame comparison module, and metadata management module. The query generation module employs natural language processing techniques to analyze configuration entries and dynamically generate SQL queries tailored to specific test cases. The data frame comparison module compares the results of different test cases using distributed collections, enabling parallel processing and efficient result comparison. The metadata management module captures and stores relevant metadata for traceability and auditing purposes. The system facilitates comprehensive validation of data pipelines, enabling organizations to ensure the accuracy, reliability, and integrity of data.

The invention discloses an agricultural crop monitoring system and method based on multi-source remote sensing data, and relates to the technical field of agricultural crop production, and the system comprises a ground remote sensing platform which is used for regularly collecting crop growth data and carrying out the recognition and prediction operation of crop growth vigor, the ground remote sensing platform comprises a crop region division unit, a data analysis unit, a data management server, a memory and a display panel; the monitoring device is used for carrying out omnibearing monitoring on the growth condition of agricultural crops, the ground remote sensing platform and the monitoring device communicate with each other through a network, and the monitoring device comprises a ground monitoring station and unmanned aerial vehicle equipment. Compared with a traditional manual monitoring mode and a single remote sensing monitoring mode, human resource cost can be reduced to a certain extent, crop growth conditions can be more comprehensively known through the unmanned aerial vehicle, the ground monitoring station and the like, the monitoring range is increased, the monitoring precision is improved, and abnormal changes in the crop growth process are found in time.

A system (100) for adaptive data blending across heterogeneous BI platforms for real-time decision making, comprising: (a) a data source abstraction module configured to establish standardised and secure connections with several different Business Intelligence (BI) platforms and convert their native data structures into a unified intermediate schema; (b) a real-time data ingestion and synchronisation module operable to continuously monitor and retrieve data updates from the BI platforms using event-driven mechanisms and configurable synchronisation policies; c) an adaptive data mapping and transformation module designed to intelligently harmonize, adapt, and semantically translate disparate data formats using rule-based logic and AI-assisted schema mapping; (d) a semantic context and metadata management module configured to maintain a unified data dictionary, track data sequence and maintain semantic consistency across platforms; (e) a cross-platform query orchestration engine designed to decompose common user queries, translate them into platform-specific query languages, execute them across the relevant BI platforms, and aggregate the results into a coherent output; (f) a decision intelligence and recommendation module that analyses the mixed data to generate real-time insights, alerts and contextual recommendations for decision support; and (g) a security, governance and compliance module configured to enforce access controls, maintain audit trails and ensure compliance with data protection regulations; h) the system provides seamless, real-time and secure data blending to enable consistent analytics and decision-making across heterogeneous BI environments.

The invention provides an intelligent power grid safety early warning system and a monitoring method in the technical field of intelligent power grids, which are used for solving the problem of safety early warning and monitoring function defects of the existing imperfect intelligent power grids, and comprise an information acquisition unit, a data management unit, an abnormity analysis unit and a safety early warning unit, the method comprises the steps of collecting power grid real-time data and line equipment data; constructing an adjacent matrix and a target matrix, calculating and analyzing to obtain a node weight and a target correlation degree, and analyzing by using a discrete model to obtain a dispersion index; a data evaluation value is obtained, and a data abnormal value is obtained through analysis; and generating intelligent power grid safety early warning information and a disaster recovery strategy according to the data abnormal value and the line equipment data. By constructing an intelligent power grid monitoring and early warning system, power grid equipment with potential safety problems is predicted in real time, the operation and maintenance efficiency is improved, the method can be suitable for power grid systems of different scales, the influence of faults on power grid operation is reduced, and large-area power failure events are effectively prevented.

A data pipeline validation system and method configured to partially automate testing of data pipelines in a distributed computing environment. The system includes a data pipeline analytic device equipped with various modules, such as a query generation module, data frame comparison module, and metadata management module. The query generation module employs natural language processing techniques to analyze configuration entries and dynamically generate SQL queries tailored to specific test cases. The data frame comparison module compares the results of different test cases using distributed collections, enabling parallel processing and efficient result comparison. The metadata management module captures and stores relevant metadata for traceability and auditing purposes. The system facilitates comprehensive validation of data pipelines, enabling organizations to ensure the accuracy, reliability, and integrity of data.

The invention provides a beach big data geographic space intelligent application service method based on a large model Agent, and relates to the technical field of geographic space data artificial intelligence processing and analysis. According to the method, a distributed platform comprising a parallel computing cluster, a spatial database and a large language model is constructed, and standardized storage and metadata management of multi-source remote sensing, vector and attribute data are supported. The system generates a spatio-temporal query instruction through natural language analysis, completes cross-source matching and spatial calculation, outputs an analysis result in a multi-modal form such as a map, a chart and a text, and realizes intelligent and interactive service of geographic data processing.

The embodiment of the invention provides a multi-modal low-altitude grid data dynamic indexing method and device for improving a cache mechanism. The method comprises the following steps: mapping multi-modal low-altitude grid data to a unified vector space, and constructing a cross-modal semantic association hybrid index for the mapped low-altitude grid data; dividing the mapped low-altitude grid data into a plurality of dynamic sub-regions according to data distribution density and data access popularity, and managing a mixed index corresponding to each dynamic sub-region by combining a Loki log index fragmentation mechanism and tagged metadata; the mapped low-altitude grid data is cached by adopting an EHCD hierarchical multi-level cache architecture; and carrying out cross-modal low-altitude grid data retrieval by combining SIMD instruction set parallelization vector similarity calculation and GPU acceleration complex space relation operation. In this way, the real-time retrieval efficiency, the system expansibility and the energy efficiency ratio of the low-altitude grid data can be remarkably improved.

The invention discloses an edge-end scene-oriented end equipment efficient enabling method, and relates to the technical field of software design and development, centralized management and control of heterogeneous equipment are realized through cloud, edge and end cooperative work, the method comprises the following steps: a cloud side constructs a unified metadata management system by using a Kubernetes technology, and provides equipment discovery, registration, management and monitoring functions; the side is responsible for access, standardized processing and analysis of equipment data, and synchronizes equipment state information to the cloud; the end side supports a plurality of Internet of Things protocols through the equipment service module and carries out data acquisition and preprocessing; according to the method, the advantages of cloud computing, edge computing and end equipment are effectively integrated, and the real-time performance and reliability of the system are improved.

The invention discloses a metadata prefetching method and device, a storage medium, electronic equipment and a program, and relates to the field of metadata management of a distributed storage system.The metadata prefetching method comprises the steps that a first information block corresponding to a metadata operation event in a target storage area of each node is obtained; updating the three-dimensional correlation degree between any two metadata in each node according to the first information group; determining a prefetching priority corresponding to any two pieces of metadata through the updated three-dimensional correlation degree; and generating a heat association list of the metadata in each node according to the prefetching priority, and respectively sending the heat association lists corresponding to the plurality of nodes to the client so as to indicate the client to prefetch the target metadata in the plurality of nodes according to the plurality of heat association lists. Through the method and the device, the problem of low pre-fetching accuracy caused by dependence on static path association of the metadata and poor cross-node adaptability of a pre-fetching method for the metadata in a distributed storage system is solved, and the pre-fetching accuracy is improved.

A data deduplication method includes: receiving a write request with first data block; writing the first data block into a storage device; writing metadata of the first data block into a first partition that is in a plurality of partitions of a metadata management structure and that is determined based on a feature of the first data block; and deleting the metadata of the first data block in the first partition and deleting the first data block from the storage device based on address information of the first data block when a fingerprint that is the same as the fingerprint of the first data block exists in the first partition. The method can prevent infrequently updated data from being evicted because resources are occupied by frequently updated data, thereby improving a deduplication ratio.

The invention relates to a metadata management system and method based on a universal metadata model of a data lake, and belongs to the technical field of metadata models, and the system comprises a metadata uptake module which uptakes metadata in the process that multi-source heterogeneous data enters the lake; the modeling module is used for modeling the acquired metadata through a metadata model and supporting dynamic expansion; the metadatabase is used for receiving and storing the ingested metadata; the metadata management module is used for managing metadata and data consanguinity of the multi-source heterogeneous data in the data lake and carrying out visual display in a graph form; wherein modeling is carried out on the acquired metadata through the metadata model, and dynamic expansion is supported; comprising the steps that metadata is modeled in a graph form, nodes are used for representing data entities and metadata entities, and a data structure is self-defined and comprises structured data, semi-structured data and unstructured data; and the incidence relation between the nodes is represented by edges. The method supports multi-source heterogeneous data cases in a data lake.

The invention provides a virtual reality simulation training-based outside-related alarm on-site disposal system, which comprises a virtual scene construction unit, a data management and analysis unit, a course design module, a resource library and an evaluation analysis system, and is characterized in that a scene editing system is constructed in the virtual scene construction unit; a landform generator and a building and facility editor are arranged in the scene editing system, and an event trigger and a behavior editor are further arranged in the virtual scene construction unit. The data management and analysis unit is used for recording data such as operation behaviors, decision-making processes and interaction conditions with virtual roles of police officers in each training, various complex, extreme or rare police affair scenes can be easily simulated, students can fully exercise the ability to cope with dangerous situations under the risk-free condition, and the training efficiency is improved. The training performance of the trainee can be accurately evaluated, the trainee can timely know the training effect of the trainee, and the instructor can conveniently adjust the teaching strategy according to the condition of the trainee.

The invention relates to an event trigger system based on an openGauss database. The system comprises: an event trigger definition module supporting a user to define a trigger rule of an event trigger; the trigger function execution module provides an execution environment for a function associated with the trigger and executes function logic; and the event monitoring and processing module monitors database operation events, matches corresponding event triggers according to trigger rules, and coordinates the execution sequence and error processing of correlation functions of the event triggers. According to the system, flexible trigger rule definition is supported through extension grammar, the metadata management efficiency is guaranteed by combining semantic verification and a special storage structure, the execution performance is optimized by utilizing pre-compiling and caching technologies, accurate monitoring and trigger coordination of events are realized through a kernel hook mechanism, and the reliability of the system is improved. The limitation that a traditional trigger only covers a DML event is effectively solved, the reliable management and control capability of full-type operation (DML / DDL) is provided for the openGauss database, and the safety, maintainability and application stability of the database are remarkably improved.

The invention discloses a metadata version control and consistency verification method and system, and belongs to the technical field of distributed system metadata management, the method comprises the steps of version tree construction, version query and backtracking, version merging and conflict processing, the version tree construction takes an initial timestamp associated with an initial version of metadata as a root node of the version tree; whenever the metadata is changed, a unique timestamp is generated for the changed metadata; carrying out associated storage on the generated timestamp and the changed metadata; and creating a new node according to the newly generated timestamp, establishing a father-son relationship between the new node and the node of the previous version to form a branch structure of the version tree, and arranging the nodes in the version tree according to the sequence of the timestamp. According to the method, the management efficiency of multi-version metadata can be remarkably improved, the data integrity and consistency are effectively guaranteed, and the method is suitable for cloud storage system metadata management, cross-platform metadata migration and other scenes.

The invention discloses an equipment instrument quality interpretation device based on AI vision, and relates to the field of computer vision and artificial intelligence. Comprising an optical acquisition system, an image processing unit, an intelligent detection unit and a data management unit, the optical acquisition system acquires image or video information of equipment and instruments through a camera or an image sensor, is matched with a corresponding light source to enable the acquired image to have no shadow, and is connected with the image processing unit through a network; the image processing unit preprocesses the acquired image; the intelligent detection unit extracts features related to target quality from the preprocessed image by using a deep learning algorithm; and the data management unit stores the data and sends an interpretation result to a user through the feedback module according to the quality interpretation result, the user performs corresponding operation according to the interpretation result, and real-time quality detection is realized by utilizing efficient image processing and AI interpretation algorithms, so that the detection efficiency is remarkably improved, and the requirements of a high-rhythm production line are met.

The invention discloses a distributed file system metadata index optimization method based on remote direct memory access, which comprises directory metadata index based on path name hash: defining a plurality of layers of directories in a directory tree as key directories, and regarding the path as a character string. After calculation of the hash function, determining the storage position of the data in the memory of the metadata server of the distributed file system; and optimizing the memory position of the file metadata based on the popularity, namely placing the file metadata which is frequently accessed at a position adjacent to the key directory metadata to which the file metadata belongs. A metadata management module in a metadata server of the distributed file system regularly checks the popularity of the file and carries out cold and hot migration according to the situation; and client cache optimization oriented to key directory metadata: a metadata cache of a distributed file system client caches the metadata and directory entries of the key directory. Optimally assuming that the metadata and directory entries of the key directory in the cache are not invalid, and updating the cache according to the result of the next-level metadata search. According to the method, the remote reading performance of the metadata index of the distributed file system is optimized by fully utilizing the characteristic of remote direct memory access, the network transmission frequency is reduced, and the load of the metadata server is effectively reduced.

The invention discloses a financial digital intelligent management system, and relates to the technical field of computers, and the system comprises an atomic power unit ACU, a side effect metadata management module which is used for decomposing a financial process into ACU and defining the structured side effect metadata of the ACU, and the structured side effect metadata comprises an influence object, a type, a condition, a range, a preset risk level and the importance of an associated data object; the dynamic process arrangement engine module is used for generating candidate execution paths according to the process definition and the ACU side effect metadata, and selecting a final execution plan by taking the score minimization as an optimization target; and the execution control and risk relief module selectively performs influence domain sandbox rehearsal before actual execution of the ACU according to a risk assessment result, and executes conditional rollback or compensation when execution fails or unacceptable side effects are generated. According to the invention, through prospective side effect management and intelligent risk avoidance and control, the stability, reliability, auditing performance and combinable innovation capability of a financial automation process are significantly improved.

Solutions for integration and ongoing management of enterprise applications operating in a networked environment are disclosed. An AI transformer extracts, transforms, and manages integration point information from various sources, including documents, UMLs, and other system diagrams. Instead of relying on traditional documentation, the invention utilizes blockchain-based NFTs (non-fungible tokens) to store and manage integration point information. NFTs are created using smart contracts, which define the rules and conditions for the NFT. A metadata engine prepares metadata for generating the Integration Point NFTs. An NFT manager creates the integration point NFTs. A change analyzer determines the impact of changes to integration points on different applications and functionalities. A release compliance component is integrated with DevOps and is responsible for ensuring that integration point changes are incorporated into all impacted applications according to the release defined in the NFT.

The invention provides an interpretable building contract risk review method and system with a mechanism layer knowledge graph, and the method comprises the steps: receiving a registered contract basis and a to-be-reviewed contract, and obtaining a standardized clause unit set; obtaining an executable mechanism library based on a large language model LLM; inputting, analyzing and normalizing values in a contract to be examined according to a mechanism, compiling and executing a mechanism formula, and obtaining a mechanism output result and a risk node aggregation probability; constructing a structured evidence packet and driving the knowledge graph to generate a risk level, a reason chain and a mending suggestion; and assembling the clause unit, the mechanism node and the risk node into a knowledge graph for exporting, and generating an interpretation path score for review and LLM reference. According to the method, the mechanism layer is introduced into the term layer and the risk layer of the knowledge graph, a causal chain between term parameters and risk triggering logic is explicitly represented, meanwhile, mechanism nodes are managed through formula and versioning metadata, and when market parameters change, an overall model does not need to be retrained or a rule base does not need to be modified on a large scale.

The invention belongs to the technical field of computer server management, and particularly discloses a laboratory collaborative environment-oriented user and data full-life-cycle server management system, which comprises a container virtualization layer, a server management layer and a server management layer, dynamic resource allocation is supported through a state saving-resource reallocation-environment recovery mechanism; the heterogeneous transaction coordination layer adopts a Saga transaction mode to coordinate a heterogeneous system so as to realize cross-system transaction consistency; the data collaboration ecological layer establishes typed management of a personal data volume, a shared software environment volume and a public data set volume, and supports software environment encapsulation sharing and versioning management; and the metadata management layer realizes real-time update and high-performance query of metadata based on a Redis hot update architecture. According to the method, the problems of inflexible resource allocation, incomplete environment isolation, dilemma in authority management and difficulty in data collaboration and recovery in the prior art are solved.

The invention provides an unstructured data lake hierarchical storage and verification system and method, and relates to the technical field of data storage and verification. The system comprises an application layer, a data lake interface layer, a metadata management layer, an unstructured data storage layer and a verification service layer, the application layer realizes data access and operation by calling services provided by the data lake interface layer; the data lake interface layer provides a unified access interface to realize unified input and output of structured and unstructured data; the metadata management layer is used for storing metadata information of user files and supporting multi-dimensional retrieval; the unstructured data storage layer is used for storing user files transmitted through the data lake interface layer, supporting various storage media and storage schemes, and automatically allocating the storage media based on a preset rule; and the verification service layer provides a data verification service. According to the method, automatic scheduling, unified access and multi-dimensional verification management of cold and hot hierarchical storage in the unstructured data lake are realized.