91 results about "Metadata service" patented technology

Novel techniques are disclosed for providing vPLC-specific metadata service including customized vPLC-specific metadata. In certain embodiments, each vPLC may generate a customized metadata using its corresponding vPLC-specific customization instructions. In some embodiments, a vPLC-specific metadata service may be performed using pre-generated customized vPLC-specific metadata, on-the-fly customized metadata, pre-generated CSP-format metadata, or combinations thereof.

The invention discloses a grouping clustering method and application of a distributed storage system. The method comprises the steps that nodes in a target cluster are grouped and belong to M sub-clusters, and M is larger than or equal to 2; respectively distributing corresponding borne services for the M sub-clusters, and limiting service access among the sub-clusters; the metadata service center of the target cluster is deployed in a metadata service sub-cluster, the metadata service sub-cluster is one of M sub-clusters, and the metadata service sub-cluster comprises a main node and a standby node for operating the metadata service center of the target cluster. According to the method disclosed by the invention, the target cluster nodes are grouped, the services are allocated to the sub-clusters, and the access among the sub-clusters is limited, so that the services can be effectively isolated in the target cluster, and the security of important information data of an enterprise is ensured. The metadata service center is deployed in a metadata service sub-cluster and comprises a main node and a standby node, so that the reliability of storage service can be ensured.

The invention provides a temporary check block repairing method and system based on erasure codes in distributed block storage, and the method comprises the steps that a distributed block storage system detects that a check block fails to be written in, and a metadata service creates a temporary check block according to an erasure code mode; the temporary check block records an operation log and executes data writing, and replays IO operation according to a data version number; repairing the inconsistency of the data blocks by using the operation log of the temporary check block; and quickly recovering the check block data through the data bitmap of the temporary check block. According to the rapid recovery processing method for the temporary check block, the check block data can be directly recovered without a coding mode, the internal IO is reduced, and the recovery speed of the erasure code check block is increased.

The invention provides a heterogeneous database query method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out security verification processing on a plurality of SQL execution requests sent by a user side, and carrying out SPI mechanism scanning processing on a plurality of database plug-ins in a heterogeneous database plug-in directory based on a high-performance database connection pool after the security verification processing is passed, determining a target database plug-in package corresponding to each SQL execution request; and performing metadata query processing on the SQL execution request in the target data source based on a database operation client interface and a metadata service interface under the target database plug-in package, and determining a structured tree result. According to the method, an extensible and dynamically integrated heterogeneous database plug-in mechanism is realized, unified access of various databases is supported, the platform maintenance cost is remarkably reduced, hierarchical metadata of any type of databases is obtained through a consistent metadata service interface, and the user experience and the cross-database query efficiency are improved.

The invention discloses a metadata space management method, device and equipment and a readable storage medium, and relates to the technical field of computers, the method comprises the following steps: selecting a target metadata service as a default metadata service; the default metadata service is used for distributing the logic space of the management storage pool; when the metadata spaces of other metadata services except the target metadata service are lower than a lower use limit threshold value, sending a space application instruction to the target metadata service; and after the target metadata service responds to the space application instruction and distributes part of logic space of the storage pool to other metadata services, the other metadata services store the distributed space as own metadata space. In this way, the metadata service manages the metadata space, management of the metadata service space of multiple nodes is achieved only through one storage pool, and no longer depends on an object storage daemon process and reliable and automatic distributed object storage.

The invention provides an object storage processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining configuration information of a target deployment region, the configuration information comprising metadata service configuration information and data storage service configuration information; corresponding metadata clients are initialized according to the metadata service configuration information, the metadata clients comprise at least two types, and each type is matched with different distributed database services; according to the data storage service configuration information, a corresponding storage rear end is initialized, the storage rear end comprises at least two types, and each type is adaptive to different object storage services; performing service registration on the initialized metadata client and the storage rear end to form an executable object storage service instance; and generating a uniform object storage service interface based on the registered object storage service instance.

A system in which an event manager executes a multitenant computing platform. The event manager obtains a first domain-specific event indicating a user request to perform operations associated with a particular tenant's domain object. Information for the domain object associated with the domain-specific event is retrieved from a tenant database, the information including a domain object definition and instructions defining its behaviors. A protected event execution environment (sandbox) is generated on the multitenant computing platform implementing restrictions on execution of the domain object's behavior instructions. The restrictions are specific to the combination of the user and the domain object. Execution of the instructions is initiated in the protected event execution environment. A second domain-specific event which indicates a request to modify the domain object information is then obtained, and the modification of the domain object information is initiated in the protected event execution environment.

The invention discloses a distributed block storage-oriented metadata request concurrency control method and system, and the method comprises the steps: firstly, carrying out the grouping of metadata according to the attribute information and business logic information of the metadata of a distributed block storage system; a metadata structure tree is established based on business logic information between root nodes of the sub-level tree structures obtained through grouping, then a tree-shaped lock mechanism is constructed based on the metadata structure tree, and fine-grained lock management is achieved; the method comprises the following steps: creating a transaction ID for a request for a time point and an RPC request type of each user RPC request, analyzing the user RPC request corresponding to each transaction ID, determining a specific node required to be processed by the request, then starting from a general control node of a tree lock, traversing all related root node paths downwards, and carrying out locking operation step by step. The availability of the metadata service is ensured, the requests on different sub-trees are allowed to be executed concurrently due to the fine-grained characteristic of the tree lock, and the concurrency and throughput of the metadata service are improved.

The embodiment of the invention discloses a metadata management method and related equipment, belongs to the technical field of distributed file systems, and is used for balancing metadata management capacity and concurrent access performance and improving metadata management efficiency. The method comprises a management node applied to a distributed file system, the distributed storage system further comprises a metadata node, and the metadata node is used for storing metadata. The method comprises the following steps: acquiring path feature information from a data warehouse metadata service, wherein the path feature information is used for representing a directory hierarchical relationship followed by a metadata node when the metadata node stores metadata; determining routing node information based on the path feature information; sending the routing node information to the client; the routing node information is used for the client to determine a target metadata node corresponding to the access request, and the access request is used for performing access operation on the target metadata node.

Novel techniques are disclosed for providing vPLC-specific metadata service including customized vPLC-specific metadata. In certain embodiments, each vPLC may generate a customized metadata using its corresponding vPLC-specific customization instructions. In some embodiments, a vPLC-specific metadata service may be performed using pre-generated customized vPLC-specific metadata, on-the-fly customized metadata, pre-generated CSP-format metadata, or combinations thereof.

The invention provides a metadata management method which is applied to a data system, the data system comprises a plurality of metadata service nodes and a metadata management node, each metadata service node is used for managing at least one metadata partition, and each metadata partition corresponds to part of metadata of data in the data system. The method comprises the steps that a metadata management node obtains a first query request, and a target metadata service node where a metadata partition corresponding to target data is located is determined; the metadata management node sends a second query request to the target metadata service node; and the first metadata service node in the target metadata service node obtains metadata corresponding to the target data in the metadata partition managed by the first metadata service node, and obtains the target data according to the metadata. An existing metadata storage and management method in a data system is low in efficiency, the method is high in metadata file access efficiency, and the load stability of the data system is good.

Embodiments of the present disclosure provide techniques for implementing a service provider to collect quantum service metadata about multiple quantum systems on multiple quantum machines and storing this quantum service metadata in a service registry. The service provider may transmit a request for access with a qubit registry to each of a plurality of quantum machines, where the plurality of qubit registries store quantum service metadata associated with a plurality of quantum services configured on the plurality of quantum machines. The service provider may receive acceptance for access to each of the plurality of qubit registries. In addition, the service provider may periodically request the quantum service metadata from the plurality of qubit registries and store the received quantum service metadata in a service provider registry.

A metadata synchronization system enables real-time interactive data exploration through a distributed metadata architecture. The system propagates metadata updates through an event-based synchronization path that maintains metadata consistency across system components. For discovered data sources, the system concurrently manages metadata in primary and secondary services instead of following traditional batch synchronization. A primary metadata service generates and manages metadata definitions while an event bus component propagates metadata update events to a secondary service maintaining a local metadata store. A query service provides immediate access to metadata for data exploration operations. In some implementations, the event bus components enables near-rime metadata availability. In some implementations, the secondary metadata service processes direct metadata updates and maintains metadata states prior to synchronization with the primary metadata service. The system reduces metadata access latency by eliminating batch synchronization overhead, enables immediate data exploration through coordinated metadata management, and maintains consistency through stateful task tracking.

A hybrid transactional / analytical processing (HTAP) database is provided, which includes an online transaction processing (OLTP) engine, an online analytical processing (OLAP) engine provided separately from the OLTP engine, and a shared storage holding data. The shared storage includes an OLTP storage holding data in row format for responding to queries of the OLTP engine, and an OLAP storage holding data in column format for responding to queries of the OLAP engine. The OLAP storage includes a delta store configured to record changes made to the OLTP storage in an update, and a base store configured to implement the update from the delta store on data in persistent storage in a flush. The HTAP database further includes a table metadata service configured to generate and perform operations on table metadata of the data that is updated in the flush.

A lakehouse metadata tag creation method, a lakehouse metadata tag query method and an electronic device are provided. The method includes: after a metadata service gateway determines a metadata tag creation request based on a metadata tag creation parameter provided by a data engine, sending, by the metadata service gateway, the metadata tag creation request to a metadata tag service system, so that the metadata tag service system executes a tag creation logic based on the metadata tag creation parameter carried in the metadata tag creation request, to implement a requirement of tagging one or more pieces of metadata.

The invention discloses a read-write method and device of a disk file, electronic equipment and a storage medium, and relates to the technical field of computers.The method comprises the steps that under the condition that a forwarding mark of a to-be-read-write file is stored in latest metadata information of the to-be-read-write file, a read-write forwarding message is constructed; the forwarding mark is cached to the latest metadata information under the condition that a plurality of clients simultaneously initiate metadata opening requests for the to-be-read-write file, and the forwarding mark represents that the read-write request for the to-be-read-write file is forwarded to the target centralized control client; the target centralized control client is a client which uniformly processes concurrent read-write requests of a plurality of clients for the file to be read and written; and forwarding the read-write forwarding message to the target centralized control client so as to perform read-write operation on the file to be read and written in the disk through the target centralized control client. Through the method and the device, the problems of slow system response time and small throughput caused by large interaction pressure between metadata service and multiple clients in the related technology are solved.

The invention relates to a cloud database computing cache system and an implementation method, and the system comprises a decentralized cache architecture module which is used for eliminating a centralized bottleneck, supporting ten-thousand-level concurrency and adapting to a big and small file mixed scene; the memory object sharing module is used for reducing intermediate result processing delay through intermediate result full-life-cycle zero-copy circulation and dynamic storage decision based on a memory pool and metadata service of the memory object sharing module; the self-adaptive asynchronous write-in module is used for connecting remote storage, multiplexing memory object sharing module resources and combining mechanisms such as delay confirmation, so that the write response speed is increased, remote requests are reduced, data integrity is guaranteed, and meanwhile, persistent support is provided for intermediate results. Through cooperative operation of the three modules, the cloud database computing cache system and the implementation method can effectively adapt to the requirements of high concurrency, large and small file mixing and elastic capacity expansion and contraction of a cloud native OLAP scene, and the integrity of data transmission and the stability of system operation are guaranteed.

The invention relates to the technical field of distributed file storage, and provides a metadata service management method and device, electronic equipment and a storage medium, and the method comprises the steps: determining a target server from a server set when the current activation number of metadata services in a cluster is smaller than a target activation number; carrying out availability check on each metadata service which is currently in a backup state on the target server, if available metadata services exist, carrying out activation, and updating the current activation quantity; and if the updated current activation number is smaller than the target activation number, continuing to determine a new target server from the server set until the updated current activation number is equal to the target activation number. According to the method, the server with the largest backup number of the currently owned metadata services is selected as the target server every time, and the available metadata services on the target server are activated, so that the load balance among the servers can be realized, and the metadata service capability with higher performance is provided.

The present application discloses a disk file reading and writing method and device, electronic device and storage medium, and relates to the field of computer technology. The method includes: constructing a read-write forwarding message when a forwarding mark of a file to be read and written is stored in the latest metadata information of the file to be read and written; the forwarding mark is cached in the latest metadata information when multiple clients simultaneously initiate metadata open requests for the file to be read and written, and the forwarding mark indicates that the read and write requests for the file to be read and written are forwarded to a target centralized control client; the target centralized control client refers to a client that uniformly processes concurrent read and write requests from multiple clients for the file to be read and written; forwarding the read and write forwarding message to the target centralized control client, so that the target centralized control client can perform read and write operations on the file to be read and written on the disk. This application solves the problem in the related art that the interaction pressure between the metadata service and multiple clients is high, resulting in slow system response time and low throughput.

The application discloses a load balancing method for distributed jobs, an electronic device and a storage medium. The load balancing method for distributed jobs is applied to a metadata service node and comprises the following steps: determining a trigger and a default execution node of the trigger meeting a preset condition, and acquiring resource usage of all execution nodes; determining a target execution node from all the execution nodes according to the resource usage, and establishing dynamic distribution information of the trigger and the target execution node in a preset global trigger execution chain table; and sending a trigger identifier of the trigger to the target execution node according to the dynamic distribution information, so that the target execution node executes a job task corresponding to the trigger. According to the embodiment of the application, the dynamic distribution of the execution node of the trigger is realized, the problem of long job execution time caused by the unbalanced load of the execution node is prevented, and the execution efficiency of the job task is improved.

This application provides a method and system for repairing temporary check blocks based on erasure coding in distributed block storage. The method includes: when the distributed block storage system detects a check block write failure, a metadata service creates a temporary check block according to the erasure coding pattern; the temporary check block records operation logs and performs data writing, replaying the IO operation according to the data version number; the operation logs of the temporary check block are used to repair data block inconsistencies; and the check block data is quickly restored using the data bitmap of the temporary check block. This method, through its rapid recovery processing of temporary check blocks, can directly restore check block data without encoding, reducing internal IO and accelerating the recovery speed of erasure-coded check blocks.

The invention relates to generation of table metadata in an HTAP database during flushing. A hybrid transactional / analytic processing (HTAP) database is provided that includes an online transactional processing (OLTP) engine, an online analytic processing (OLAP) engine disposed separately from the OLTP engine, and a shared storage that holds data. The shared storage device includes an OLTP storage device that holds data for responding to a query of the OLTP engine in a row format, and an OLAP storage device that holds data for responding to a query of the OLAP engine in a column format. The OLAP storage device includes: an incremental storage region configured to record a change made to the OLTP storage device at the time of update; and a base storage area configured to implement updates from the incremental storage area on the data in the persistent storage device during flushing. The HTAP database also includes a table metadata service configured to generate table metadata of the data that is updated at the time of flushing and perform an operation on the table metadata of the data that is updated at the time of flushing.

The present application discloses a data management method, apparatus, device and storage medium, relating to the field of distributed storage clusters, including: receiving data change information uploaded by a metadata service cluster; generating a hierarchical change table based on the data change information and generating a full-scale table based on all the data information stored in the data search and analysis engine; receiving a retrieval request sent by a user, and performing a retrieval in the hierarchical change table according to a preset retrieval method based on the retrieval request to obtain a retrieval result; performing a retrieval in the full-scale table based on the retrieval result to obtain target information corresponding to the retrieval request; and returning the target information to a web page for the user to view. In the present application, the data search and analysis engine constructs a database table through the received data change information and retrieves in the database table according to the user's retrieval request, improving the data retrieval efficiency and the competitiveness of the product.

Techniques are provided for maintaining and utilizing a file index and a file version index. Metadata may be evaluated to identify constant attributes and modifiable attributes of files. A file index of a file catalog may be populated with the constant attributes. A file version index of the file catalog may be populated with the modifiable attributes as file versions of the files. In response to receiving a request for a file, the file index and the file version index are evaluated to identify a location of the file within a data source. Access to the file at the location within the data source is provided.

The present invention provides a distributed metadata remote asynchronous replication method, apparatus, and device for addressing the technical problem of low synchronization efficiency. Based on directory anchoring technology, the present invention attributes metadata modifications to the asynchronous remote replication feature directory to an anchored metadata service node. The metadata log for the asynchronous remote replication feature directory is read from the anchored metadata service node, and metadata modifications are extracted from the metadata log. After semantic conflict resolution, concurrent real-time synchronization is performed. This invention improves the synchronization efficiency of feature directory metadata.

The invention provides a data query method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of big data. The method comprises the following steps: receiving a user query request; when target metadata corresponding to the user query request is queried in the active cache database, obtaining the target metadata from the active cache database; when the target metadata is not queried in the active cache database and the target metadata is queried in the engine database through the query engine, obtaining the target metadata from the engine database; when the target metadata is not queried in the active cache database and the target metadata is not queried in the engine database through the query engine, the query engine requests the metadata management system to pull the target metadata in the data lake; according to the embodiment of the invention, the quasi-real-time query data is provided for the business party on the premise of ensuring the query performance, the freshness of the data is ensured, and the pressure on the metadata service of the data lake is reduced.

The invention relates to a data processing method of a data lake, computer equipment, a storage medium and a product, and relates to the technical field of big data. The method comprises the steps that a data file is written into a data lake through a first calculation instance, and metadata information of the data file is generated through the first calculation instance; the first calculation instance is any one of at least two calculation instances contained in a distributed cluster; carrying out persistent processing on metadata information of the data file through the shared storage component and storing the metadata information; the shared storage component is used for providing services for at least two computing instances. By means of the method, the risk resistance of system data can be improved, even if the metadata service is different or needs to be restarted, due to the fact that the metadata information is subjected to persistent processing, the data cannot be lost, and the operation of entering a lake of the data file can be normally conducted, the stability of the system is improved, and the user experience is improved. And the timeliness of data processing is ensured.

The embodiment of the invention provides a lake and warehouse integrated data processing system, and relates to the field of big data processing. A data access layer accesses data from a plurality of heterogeneous data sources; the object storage unit is used for persistently storing the initial data accessed from the data access layer in an original data format to form an original data area; the core storage engine unit adopts a large-scale parallel processing architecture and manages a refined data area through an incremental data snapshot mechanism; the unified metadata service manages metadata in a unified manner, and records a derivative relationship and structure mapping; the data processing layer extracts data from the original data area, performs conversion and loading operation, writes the processed data into the refined data area, and is integrated with a data quality verification module; and the data service layer provides a uniform data access interface and can query and analyze data in the original data area and the refined data area. The problems of data islands, performance bottlenecks, treatment difficulty and the like in the prior art are effectively solved.

The application provides a Lustre file system write processing method and device, the method comprising: in response to a write data request initiated by an application layer for a target file, detecting whether an accelerated write condition is met; if the detection result is yes, copying to-be-written data to a preset cache area in a kernel state of a Lustre client and returning write success to the application layer, applying a distributed write lock corresponding to a file offset range of the to-be-written data to a metadata service by using a background thread, and after holding the lock, writing the to-be-written data in batches into a page cache; if the detection result is no, writing the to-be-written data into the page cache page by page, including applying a distributed write lock corresponding to a file offset range of the page data to the metadata service by using a background thread, and after holding the lock, writing the page data into the page cache; and persistently storing data in the page cache to an object storage server. Compared with related technologies, the application avoids multiple lock applications and improves data write efficiency.

A strongly consistent distributed data storage system comprises an enhanced metadata service that is capable of fully recovering all metadata that goes missing when a metadata-carrying disk, disks, and / or partition fail. An illustrative recovery service runs automatically or on demand to bring the metadata node back into full service. Advantages of the recovery service include guaranteed full recovery of all missing metadata, including metadata still residing in commit logs, without impacting strong consistency guarantees of the metadata. The recovery service is network-traffic efficient. In preferred embodiments, the recovery service avoids metadata service downtime at the metadata node, thereby reducing the impact of metadata disk failure on the availability of the system. The disclosed metadata recovery techniques are said to be “self-healing” as they do not need manual intervention and instead automatically detect failures and automatically recover from the failures in a non-disruptive manner.