210 results about "Metadata management" patented technology

The invention relates to the technical field of big data storage, and discloses a multi-level data storage and mixed query method and system, a storage medium and equipment, and the method comprises the steps: collecting original buried point data, carrying out real-time stream processing, storing the processed real-time data in a hot storage layer, carrying out timing offline batch processing on historical buried point data, and storing the processed real-time data in a hot storage layer; converting into an optimized storage format and storing in an object storage system; an extended metadata management system is constructed, and structure information, partition information and access frequency information of data in the object storage system are automatically extracted and managed; executing query, and reducing the data scanning amount through a query optimization strategy; automatically migrating data among different storage layers through a dynamic cold and hot data layering mechanism; a unified query interface is provided, a query request is distributed to a corresponding real-time processing system or an offline query system based on an intelligent routing strategy, and a query result is cached and returned, so that high-performance and low-cost storage and second-level query of data are realized.

The invention discloses a distributed block storage method and device, electronic equipment and a storage medium. The load index is calculated by collecting the multi-dimensional performance indexes of the storage nodes in real time, and the data fragment storage position is dynamically adjusted according to the load. The self-adaptive multi-dimensional load sensing model can accurately adapt to various service scenes, real-time balance of loads between nodes is achieved, hot nodes are effectively avoided, and the overall performance of the system is improved. And when the load index exceeds the preset threshold value, hot data identification, node affinity evaluation and migration income calculation modes are started to perform data fragmentation migration, so that the utilization rate of mixed medium resources is improved, a high-performance storage medium is reasonably utilized, and the data access efficiency is improved while the storage cost is reduced. Through cooperative work of load monitoring, fragmentation decision making, migration execution, metadata management and the like, a fragmentation strategy is automatically optimized according to input and output characteristics of different service types, and the overall performance under different service scenes is improved.

The invention discloses a Text2SQL (Structured Query Language) system and method based on cue words. The system comprises a natural language input module for receiving and preprocessing a natural language query statement; the metadata management module is used for managing database metadata information; the cue word generation module is used for generating cue words containing database modes, user problems and SQL requirements on the basis of the query statements and the metadata; the large language model module is used for generating a preliminary SQL based on the cue word; the SQL grammar verification module is used for verifying grammar and feeding back errors; the manual auditing module is used for auditing and editing the SQL statements passing the verification; and the SQL execution module is used for executing the SQL passing the audit and returning a result. According to the method, the accuracy is improved by combining cue word guidance and a large model, the field adaptability is enhanced by relying on metadata, the data security is ensured by introducing manual auditing, the accuracy and security of SQL generation in a complex query scene are remarkably improved, and the development and maintenance cost of the system is reduced.

The invention discloses a distributed file management system and method, and belongs to the technical field of computer software, the system adopts a tree hierarchical structure to organize files, and the system comprises an application layer used for receiving file operation requests from a plurality of independent clients, and each request carries a unique application identifier; the file system layer comprises a configuration module, an operation module, a metadata management module, a day module, a timed task module and a gateway module, and the request is routed to a corresponding logic storage partition in the data layer according to an application identifier; the data layer comprises a database cluster and an object storage cluster, and performs persistent storage and management of logic isolation on metadata and file data according to application identifiers; and the infrastructure layer provides an extensible server and network resources. Based on a distributed architecture, load balancing and efficient access are realized through data storage partitioning, a metadata management and index technology and a client load balancing technology, and the expandability, fault tolerance and performance of the system are improved.

The invention provides a storage device, a storage method and a storage access method based on lake and warehouse integration. The storage device comprises a storage management unit, a unified metadata management unit and a federal query engine unit, the storage management unit is used for realizing hybrid storage and dynamic hierarchical storage; the unified metadata management unit is used for metadata atlas construction, concurrency control and GDPR compliance support; the federal query engine is used for intelligent routing decision, SQL dialect unification and intelligent push-down optimization; based on a lake and warehouse integrated storage device and an access technology thereof, unified management of structured and unstructured data is supported, the problem of data islands is avoided, and cross-engine data visibility is realized through unified metadata management; according to the lake and warehouse integrated storage device and the access method, technologies such as a dynamic hierarchical storage strategy, intelligent routing decision, intelligent push-down optimization and the like are fused, and the storage and access efficiency and the robustness of the system are greatly improved.

The invention relates to the field of distributed storage, and discloses a distributed memory system based on erasure codes and a dynamic repair mechanism. Comprising a data node cluster, a metadata management cluster, a bandwidth token management module, a dynamic repair composer and a repair agent node. And the data node cluster divides the object into strip generation data fragments and verification fragments and dispersedly stores the data fragments and the verification fragments according to a placement rule. And the metadata management cluster maintains object stripe mapping, a fragment position set, erasure code protection configuration and a version epoch, and records a repair intention log. And the bandwidth token management module grants repair execution permission according to the fault domain budget. The dynamic repair composer generates availability repair and durability repair tasks, the temporary check fragments are written in when the number of the available check fragments is lower than a threshold value in the availability stage, and the configuration fragments are recovered and the temporary fragments are recovered in the durability stage. And the consistency of version epochs is verified before the agent node is repaired and written back, and if not, recalculation is carried out.

The invention discloses a distributed medical image retrieval and hierarchical storage management system and method. The system comprises an interface processing module, a distributed image storage cluster, a routing forwarding module, a metadata management module and a storage management module. The interface processing module receives the image retrieval request, generates a routing identifier for positioning the storage position of a target medical image file and sends the routing identifier to the routing forwarding module; and the routing forwarding module queries a mapping relationship between the medical image file and the storage node and / or the storage hierarchy based on the routing identifier, determines a first storage node and forwards the request, so that the first storage node returns a target medical image file. And the storage management module obtains the access statistical information, determines a target storage level and / or a target storage node, migrates the medical image file, and triggers updating of the mapping relation after migration is completed. Therefore, cross-node accurate positioning and unified retrieval are realized, positioning consistency is kept after file migration, and hierarchical storage management based on access conditions is supported.

The invention belongs to the field of artificial intelligence, particularly relates to an integrated artificial intelligence analysis and modeling platform supporting large-scale data processing, and aims to solve the problems of low resource scheduling efficiency, model training reasoning splitting, data algorithm coupling, limited cross-modal modeling and the like under high-dimensional heterogeneous mass data. The platform comprises a distributed data access layer, a unified metadata governance engine, a heterogeneous computing resource scheduling center, a model life cycle management module, a multi-modal fusion modeling framework and a dynamic service orchestration interface, and realizes automatic data access governance, dynamic resource scheduling, model full-cycle management and control and cross-modal intelligent fusion. And the resource utilization rate, the modeling accuracy and the service deployment efficiency are remarkably improved.

The invention relates to the technical field of computer data writing, in particular to a computer data writing system and method.The computer data writing system comprises a multi-level storage module, a data receiving and analyzing module, a heat evaluation and layering module, a differential writing module, a dynamic migration module, a metadata management module and a monitoring and scheduling module; according to the corresponding method, a hot, warm and cold three-level storage architecture is constructed, an initial data storage hierarchy is determined based on a hot degree prediction model, and differential write-in operation is executed; establishing a'migration forecast-write optimization 'cooperation mechanism, a'write performance data-storage hierarchy parameter' closed-loop optimization mechanism and a'multi-scene write strategy-hierarchy-migration 'parameter mapping system to ensure data access transparency; the cold data storage cost is reduced while the high write-in performance of the hot data is ensured, the problem that the storage performance and the cost are difficult to balance in a time sequence big data scene is solved, and the method is suitable for high-frequency time sequence data storage scenes such as power monitoring and financial transactions.

Disclosed is an improved approach to manage hierarchical metadata for a database system. A hierarchical metadata structure pertaining to a hierarchical object structure of a multitenant database architecture for a plurality of tenants may be maintained where the multitenant database architecture comprising a container database (CDB) that includes a pluggable database (PDB). A request for an access to a metadata object in the hierarchical metadata structure for one or more database objects in the container database may be received. In response to the request for the access to the metadata, access to at least a portion of the hierarchical metadata structure is provisioned.

The invention provides an optimization method and system for improving the file real-time access speed of an electric power embedded operating system, and belongs to the technical field of electric power embedded systems. The method comprises the following steps: establishing a dynamic preloading mechanism based on the operation state of the power equipment; the dynamic preloading mechanism triggers file preloading operation by monitoring the working condition change of the equipment; designing an improved cache replacement algorithm; the improved cache replacement algorithm is used for dynamically adjusting a cache strategy in combination with power file access characteristics; constructing a multi-dimensional file access scheduling system; the multi-dimensional file access scheduling system allocates system resources according to task priorities; executing a file access full-process acceleration strategy; the file access full-process acceleration strategy comprises metadata management, data prefetching and version control. The file access delay can be shortened, and the system resource utilization rate can be improved.

The invention provides a mixed storage and unified query layer data retrieval system based on an AI cloud desktop, which belongs to the technical field of AI intelligent application and comprises a unified query access layer, an SQL optimizer, a vectorization execution engine, a mixed storage engine and a metadata management center. Column storage, inverted index and row storage are organically combined, services are provided for the outside through a unified SQL query layer, and intelligent routing and vectorization computing technologies are introduced. The method is characterized in that an optimal storage engine is automatically selected for execution according to a query mode, the problems that a single inverted index architecture is low in efficiency and high in resource consumption under the scenes of ad-hoc analysis, high-concurrency point search and large-scale aggregation are solved, and better comprehensive query performance, lower data storage cost and higher system expansibility than traditional Elasticsearch are achieved.

The invention relates to an intelligent file management system based on RAG and metadata full-life-cycle management, which is characterized by comprising a front-end application layer, a back-end service layer, an RAG knowledge base analysis layer and a data storage layer, and all the layers work cooperatively through standardized interfaces; the RAG knowledge base analysis layer is constructed by taking a self-built RAG knowledge base as a core, and the back-end service layer encapsulates a metadata management center. According to the method, through a multi-dimensional retrieval mode of'metadata screening + semantic retrieval + knowledge graph association ', the retrieval range is narrowed by metadata, then a knowledge unit is accurately positioned through RAG semantic retrieval, and finally associated document information is supplemented, so that compared with traditional full-text retrieval, the retrieval precision is improved, and the retrieval response time is shortened to millisecond level; automatically mining a document association relationship through a knowledge graph construction module, and supporting a user-defined association rule; and through a document-knowledge integrated view and a scenarized knowledge aggregation function, cross-department and cross-project knowledge rapid circulation is realized, and the knowledge reuse efficiency is improved.

The invention provides a multi-source data integration method, a storage medium, a program product and electronic equipment, and can be applied to the field of financial science and technology. The method comprises the following steps: calling a corresponding adapter to read original data from each data source according to the types of a plurality of data sources; performing quality evaluation on the original data by adopting a random forest algorithm, determining to-be-cleaned data in the original data, and performing data cleaning on the to-be-cleaned data to obtain cleaned data; standardizing the cleaned data and uncleaned data in the original data by adopting a metadata management technology; under the condition that data conflicts exist in the standardized data, conflicting data are arbitrated so as to determine real data from the conflicting data, and the data conflicts represent that logic conflicts exist in data information from different data sources. The problem that a multi-source data integration scheme in the prior art is low in integration quality is solved.

The invention discloses a large model localization deployment and security isolation device for a hospital closed network environment, which consists of a model deployment unit, a security isolation unit, a data management unit, a security control unit and an intelligent updating unit, and realizes independent operation of a large model in a hospital. According to the system, safe input of medical data is achieved through data cleaning, desensitization and formatting processing, full-process safety of data and model files is guaranteed through authority control, access recording and encrypted storage, and credible upgrading and abnormal rollback of a model are completed through a one-way isolation and offline updating mechanism. The large model can be operated and maintained under the condition that an external network is not connected, the requirements of hospitals for data privacy and network security are met, and the method is suitable for various types of intelligent medical services.

The invention relates to a software and hardware combined fine-grained memory protection mechanism, which realizes high-speed mapping from a physical address to a fine-grained permission label by integrating a metadata search unit and a metadata conversion lookup buffer on a critical path of a processor loading / storage unit. An operating system maintains a multi-level fine-grained permission metadata table in a main memory, and the minimum memory protection granularity is refined to a 64-byte sub-page level. When a processor executes a memory access instruction, address conversion and permission verification are completed in parallel, an access type and a permission label are compared in real time within 1-2 clock periods, and when permission conflicts are detected, high-priority abnormity is triggered immediately, and illegal addresses and fault types are reported accurately. The method supports instruction set extension, buffer overflow protection, multi-level metadata management and user mode and kernel mode differentiated authority control, reduces the influence on the performance of the processor while improving the security of the memory, and is suitable for a computing system with high security and high performance.

A method and system for identifying IP addresses of an edge device, obtaining associated IP metadata, and storing both the IP address and IP metadata locally is disclosed. The method comprises receiving, by a server, a network message from an edge device that includes an Internet Protocol (IP) address of the edge device, wherein the edge device is not privy to its own IP address. The server extracts the IP address from the network message and obtains associated IP metadata from an IP metadata server. The IP metadata is then transmitted from the server to the edge device for local storage, after which the server deletes the IP metadata to ensure it is not persistently accessible to the server while remaining persistently accessible to the edge device. The edge device utilizes the locally stored IP metadata in executing predefined functionality. The invention also encompasses a system comprising a server, metadata server, and edge device configured to perform these operations. Both the method and system aspects enable efficient IP metadata management while addressing privacy concerns and reducing server-side storage requirements through distributed edge-based storage architecture.

The invention provides a device for automatic metadata management and data knitting treatment, and the device comprises a semantic mounting unit which is used for carrying out the semantic tag mounting of a structured metadata set of a data factory based on a semantic model of a data knitting architecture, so as to generate enhanced semantic metadata and a complete data line record; the rule binding unit is used for carrying out governance rule binding on the enhanced semantic metadata and the complete data line record on the basis of a data knitting federation governance rule so as to generate a metadata governance list bound with the governance rule; the change updating unit is used for synchronously updating a data set state field and a flow path recorded by a corresponding complete data line to a metadata governance list based on a change data capture mechanism when it is monitored that a data factory has a data set change, so as to generate unified metadata caalog; the interface configuration unit is used for configuring a metadata retrieval interface and a governance rule execution engine for the unified metadata caalog so that the unified metadata caalog can be accessed in the data weaving environment.

The invention relates to the technical field of big data, provides a metadata management method and device based on semantic feedback, equipment and a medium, is applied to financial and medical health care service scenes, and can reinforce a learning mechanism to initialize a state space, an action space and a reward function based on semantic feedback. The limitation that traditional reinforcement learning only depends on technical indexes is broken through, and alignment of technical optimization and business value is achieved; a metadata autonomous scheduling engine is utilized to generate a current state, a current action is selected from an action space by utilizing a current strategy, and a strategy executor is called to execute the current action, so that a global optimal scheme can be generated in combination with a running state and business semantics, and a local optimization trap is avoided; and in a preset time period, calling a metadata autonomous scheduling engine to perform strategy optimization according to the experience playback buffer area, and executing the next round of metadata management according to a new strategy, so that continuous optimization of the strategy is realized, and long-term change of a service mode can be self-adapted.

This invention discloses a distributed data query method and system for ubiquitous storage. The method comprises the following steps: Step 1, constructing a hierarchical index network model, which includes a metadata management network and a data block index network: a decentralized metadata management network is constructed using a fully connected networking approach; a data block index network is constructed based on a distributed hash table; Step 2, defining a hierarchical data index interaction paradigm to achieve collaborative management and fast querying of metadata and data blocks. This invention, through the hierarchical index network model, decouples metadata from location information, reducing the management and maintenance costs of metadata.

The invention discloses a duplication data deletion metadata management method based on a persistent memory. The method comprises the following steps: managing a global fingerprint table and a logic address to a fingerprint mapping cache in a DRAM (Dynamic Random Access Memory), and managing an additional redo log and a global mapping table in the persistent memory. The write-in path firstly writes metadata update into the DRAM and forms a transaction record, the transaction record is sequentially added to a persistent memory log in a group submission mode, persistence is aligned according to the read-write granularity of the persistent memory, and random write and lock competition is reduced; in the checkpoint stage, an aggregation strategy is dynamically selected according to the redundancy rate and the coverage write proportion obtained through online load detection, and therefore metadata write amplification is remarkably reduced on the premise that collapse consistency is guaranteed. Meanwhile, three self-adaptive deduplication modes are provided, fingerprint management is bypassed in the low redundancy stage, redundancy is sampled and estimated in the medium redundancy stage, and full deduplication is started in the high redundancy stage, so that the metadata management cost is stably reduced under different loads, and the system throughput is improved.

The application discloses a kind of high-resolution remote sensing image distributed processing system, the system includes remote sensing image input file format extension module, remote sensing image metadata management and optimization module and the distributed computing module of remote sensing image.The beneficial effects of the present application are that the scalability of distributed storage model can be used to store remote sensing images in different nodes, and as the number of storage nodes increases, the storage capacity of the image also increases accordingly.Using the corresponding distributed computing framework, the "computing moves to data" is realized, and the remote sensing image data stored in different nodes is operated simultaneously, which greatly improves the processing efficiency of image data.

The invention provides a method and system for generating an index library in a rapid configuration mode based on a domain-specific language, and relates to the technical field of computer software. Index configuration information, input by a user in a configuration interaction interface, of a target index is received; performing lexical analysis, grammatical analysis and semantic verification on the index configuration information to obtain target configuration information; according to the target configuration information, generating an executable code which can be identified by the target calculation engine, wherein the executable code is an SQL query statement; scheduling the executable code to a target calculation engine for execution, extracting data from the associated data source and completing calculation to obtain a target index, and storing the target index to a preset index library; the target index is registered to the index library center for metadata management, and an index query and data service interface is provided externally, so that efficient configuration generation and flexible iteration of the index library can be realized while the accuracy and controllability of the generation result are ensured, and the technical threshold is reduced and the existing data assets are reused.

Provided is a content addressable storage (CAS) system that allows a user to request, either through an application server or directly to one or more CAS servers, files and content related to a query. In some embodiments, the content can be discovered by searching previously-stored metadata related to each file at the content addressable storage server. The search can also be replicated across multiple content addressable storage servers in order to obtain varied results and redundant results. Duplicate results may be flagged or omitted, ˜d the results are returned to the requester.

A metadata synchronization method, apparatus, and system. The method is applied to a distributed file system. The system comprises a plurality of nodes, each node storing file system metadata, wherein a first node operates on a file and updates first metadata. The first node sends a first metadata update request to other nodes by means of a collision-free data synchronization protocol. The first node receives a second metadata update request sent by a second node on the basis of the collision-free data synchronization protocol, and the first node updates second metadata on the basis of the request. In the method, a distributed metadata management mode is used, thereby avoiding the problem in centralized management that a metadata management node is highly loaded and easily becomes a system bottleneck. The distributed management mode reduces the risk of a single point of failure, and facilitates system scalability and reliability. Metadata synchronization is performed on the basis of the collision-free data synchronization protocol, thereby ensuring data consistency, avoiding the problem of high delay caused by a complex election mechanism and a communication process, and improving synchronization efficiency.

The invention discloses an artificial intelligence multispectral sensing mobile illegal parking automatic evidence obtaining device and method, and belongs to the technical field of intelligent traffic management. The equipment integrates a multispectral sensing unit, a positioning and posture sensing unit, an edge computing unit, a data management and interaction unit and a mobile platform. The system has the beneficial effects that the law enforcement coverage range is expanded through the mobile platform, the detection limitation of severe environments such as night, rain and fog is overcome by means of a multispectral sensing technology, and the full-process automation of vehicle detection, illegal parking judgment and automatic evidence obtaining is realized by utilizing an artificial intelligence algorithm; the problems that a traditional fixed electronic police is limited in coverage range and low in manual patrol efficiency and evidence obtaining of a single sensor is restricted by the environment are solved, all-directional, all-time and automatic illegal parking accurate evidence obtaining is achieved, and the traffic law enforcement efficiency and the management level are remarkably improved.

A metadata synchronization system enables real-time interactive data exploration through a distributed metadata architecture. The system propagates metadata updates through an event-based synchronization path that maintains metadata consistency across system components. For discovered data sources, the system concurrently manages metadata in primary and secondary services instead of following traditional batch synchronization. A primary metadata service generates and manages metadata definitions while an event bus component propagates metadata update events to a secondary service maintaining a local metadata store. A query service provides immediate access to metadata for data exploration operations. In some implementations, the event bus components enables near-rime metadata availability. In some implementations, the secondary metadata service processes direct metadata updates and maintains metadata states prior to synchronization with the primary metadata service. The system reduces metadata access latency by eliminating batch synchronization overhead, enables immediate data exploration through coordinated metadata management, and maintains consistency through stateful task tracking.

The application discloses an unmanned aerial vehicle data processing system based on edge computing and belongs to the technical field of unmanned aerial vehicle data processing.The system comprises an unmanned aerial vehicle end, an edge computing node and a ground control center; wherein the unmanned aerial vehicle end is provided with a multi-modal sensor array and is configured with a data synchronization module and a preprocessing module; a pipeline processing engine is based on a Kafka+Spark framework to realize data sharding parallel processing; a dynamic unloading controller is used to schedule computing tasks according to the computing power state of the unmanned aerial vehicle; and the ground control center is configured with a multi-modal storage system, a metadata management engine and an end-to-end security channel; through the edge node localization processing and the dynamic task unloading mechanism, in combination with the pipeline parallel engine, the end-to-end data processing delay is reduced to below 500 ms; the millisecond-level synchronization of the multi-modal sensor and the adaptive compression strategy improve the data transmission efficiency, effectively solve the massive data transmission bottleneck and meet the high-time efficiency demand of real-time monitoring, emergency response and other scenes.

The invention discloses a management method of a shared directory under a distributed file system and electronic equipment, and relates to the technical field of distributed file systems, and the method comprises the following steps: creating a partition identifier when a protocol layer creates the shared directory; the file system layer creates a metadata management object of the shared directory and binds the metadata management object with the partition identifier; and determining a first target thread and a target metadata management object corresponding to the directory operation request, and processing the directory operation request according to the first target thread and the target metadata management object. The problem that when the shared directory is exported or accessed, export or access of all the shared directories is blocked due to blocking of a certain request, and access of an upper-layer application to the shared directory is affected is solved. According to the method, metadata partition management is carried out on shared directories by utilizing metadata management objects, a first target thread is set for a target shared directory to process a directory operation request, resource management and request access isolation among different shared directories are realized, and access to other shared directories is not affected by blockage of a certain request.

The invention discloses a smart park articulated naturality web node data distributed storage system, and relates to the technical field of smart park data storage. Comprising a data acquisition layer, a distributed storage node cluster, a metadata management module, an intelligent scheduling module, a data security guarantee module, a data verification module, a data adjustment module and an application interface layer, and all the modules cooperate to realize efficient storage, quality control and management of articulated naturality web node data. Aiming at a smart park articulated naturality web data storage pain point, efficient storage and management and control are realized through multi-module cooperation, a distributed architecture, mixed storage and intelligent scheduling are adopted, the storage efficiency and the resource utilization rate are improved, multi-copy backup, cross-physical deployment and automatic fault repair are realized, data reliability is guaranteed, and the system is suitable for large-scale popularization and application. The method comprises the steps of full-process encryption, authority control and multi-dimensional verification, building of a data security defense line, data dynamic optimization and standardization of an interface, enhancement of data availability and system compatibility, and adaptation to the dynamic increase demand of park data.