112 results about "Metadata service" patented technology

Novel techniques are disclosed for providing vPLC-specific metadata service including customized vPLC-specific metadata. In certain embodiments, each vPLC may generate a customized metadata using its corresponding vPLC-specific customization instructions. In some embodiments, a vPLC-specific metadata service may be performed using pre-generated customized vPLC-specific metadata, on-the-fly customized metadata, pre-generated CSP-format metadata, or combinations thereof.

The invention discloses a grouping clustering method and application of a distributed storage system. The method comprises the steps that nodes in a target cluster are grouped and belong to M sub-clusters, and M is larger than or equal to 2; respectively distributing corresponding borne services for the M sub-clusters, and limiting service access among the sub-clusters; the metadata service center of the target cluster is deployed in a metadata service sub-cluster, the metadata service sub-cluster is one of M sub-clusters, and the metadata service sub-cluster comprises a main node and a standby node for operating the metadata service center of the target cluster. According to the method disclosed by the invention, the target cluster nodes are grouped, the services are allocated to the sub-clusters, and the access among the sub-clusters is limited, so that the services can be effectively isolated in the target cluster, and the security of important information data of an enterprise is ensured. The metadata service center is deployed in a metadata service sub-cluster and comprises a main node and a standby node, so that the reliability of storage service can be ensured.

The invention relates to an agent method and device for a vertical sub-database, equipment and a medium. The method comprises the following steps: when a database creation instruction of a user is received, querying a target routing strategy corresponding to the database creation instruction; selecting a target fragment from the created fragments according to a target routing strategy; executing the database creation instruction on the target fragment to create a logic database, and generating a creation log of the logic database; after the logic database is created, generating a mapping relation between the logic database and the target fragment; the mapping relation and the creation log are stored in metadata fragments, the metadata fragments are one fragment in the created fragments, and the mapping relation is a routing rule of an instruction for operating the logic database. According to the method and the device, the problems of data skew, data migration difficulty and the like caused by routing rules can be avoided, additional metadata service storage and management of the metadata do not need to be introduced, and storage and management of the metadata are more convenient.

The invention belongs to the field of big data, and relates to a directory path analysis method, which comprises the following steps: acquiring directory information of a file system, and creating a metadata cache in a metadata client for storing the directory information; receiving a directory path analysis request, and judging whether a directory accessed by the directory path analysis request exists in the metadata cache or not based on the directory information; if the directory accessed by the directory path analysis request exists in the metadata cache, acquiring an entity identifier corresponding to the directory from the metadata cache; sending the entity identifier to a metadata server, so as to query result data corresponding to the directory from a key value database through the metadata server; and returning result data corresponding to the directory path analysis request. The invention further provides a device, equipment and a medium. In addition, the invention also relates to a block chain technology, and directory information and result data can be stored in a block chain. According to the method, the directory path analysis time delay can be reduced, and the problem of near root hotspots is solved.

The invention provides a temporary check block repairing method and system based on erasure codes in distributed block storage, and the method comprises the steps that a distributed block storage system detects that a check block fails to be written in, and a metadata service creates a temporary check block according to an erasure code mode; the temporary check block records an operation log and executes data writing, and replays IO operation according to a data version number; repairing the inconsistency of the data blocks by using the operation log of the temporary check block; and quickly recovering the check block data through the data bitmap of the temporary check block. According to the rapid recovery processing method for the temporary check block, the check block data can be directly recovered without a coding mode, the internal IO is reduced, and the recovery speed of the erasure code check block is increased.

A central node can: receive a query comprising at least one parameter comprising a time range of a dataset stored in a cloud storage system; transmit one or more of the query parameters comprising the time range to a metadata service; receive from the metadata service a list of files related to the query; and assign to each processing node of a plurality of processing nodes a subset of the files. Each processing node can: determine that the subset is not stored on a cache; retrieving the subset not stored on the cache from the cloud storage system; store the retrieved subset in a local memory; scan the subset stored in the local memory for data matching the at least one parameter to generate a subset of query results; and concurrently copy using a separate thread from the scanning, the subset stored in the local memory to the cache.

A computer-implemented method of caching multi-session data communications in a computer network, includes the steps of: (a) receiving, intercepting, or monitoring one or more data sessions between a client executing a multi-session application for retrieving a desired content object and one or more metadata services, said client communicating with the one or more metadata services to discover metadata for the content object; (b) analyzing queries and responses exchanged between the client and the one of more metadata services to discover metadata for the content object; (c) receiving or intercepting subsequent data sessions between the client and content sources; (d) identifying a data protocol used by the client and identifying data queries within the data sessions; (e) identifying the content object or portions thereof requested by the client in the data queries; and (f) determining if the content object or portions thereof are stored in cache and, if so, sending the content object or portions thereof stored in cache to the client, and, if not, sending the data queries to the content sources, storing data responses from the content sources, and sending the data responses to the client.

The invention provides a heterogeneous database query method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out security verification processing on a plurality of SQL execution requests sent by a user side, and carrying out SPI mechanism scanning processing on a plurality of database plug-ins in a heterogeneous database plug-in directory based on a high-performance database connection pool after the security verification processing is passed, determining a target database plug-in package corresponding to each SQL execution request; and performing metadata query processing on the SQL execution request in the target data source based on a database operation client interface and a metadata service interface under the target database plug-in package, and determining a structured tree result. According to the method, an extensible and dynamically integrated heterogeneous database plug-in mechanism is realized, unified access of various databases is supported, the platform maintenance cost is remarkably reduced, hierarchical metadata of any type of databases is obtained through a consistent metadata service interface, and the user experience and the cross-database query efficiency are improved.

The invention discloses a metadata space management method, device and equipment and a readable storage medium, and relates to the technical field of computers, the method comprises the following steps: selecting a target metadata service as a default metadata service; the default metadata service is used for distributing the logic space of the management storage pool; when the metadata spaces of other metadata services except the target metadata service are lower than a lower use limit threshold value, sending a space application instruction to the target metadata service; and after the target metadata service responds to the space application instruction and distributes part of logic space of the storage pool to other metadata services, the other metadata services store the distributed space as own metadata space. In this way, the metadata service manages the metadata space, management of the metadata service space of multiple nodes is achieved only through one storage pool, and no longer depends on an object storage daemon process and reliable and automatic distributed object storage.

The embodiment of the invention provides a data access method and device, a storage medium and electronic device.The data access method comprises the steps that container service is started in each controller, and a metadata service component and a file sharing protocol service component are added in a container mirror image managed by each container service, each controller is a controller in a disk array; under the condition that a data access request sent by target equipment is received, accessing target file data through a first container service in a first controller and a first block service in the first controller, the first container service and the first block service perform data interaction through a first shared memory in the first controller, and the data access request is a request for accessing target file data corresponding to a first disk space managed by the first controller.

The invention provides a file lock processing method and device, electronic equipment and a storage medium. Under the condition that a file stream client obtains a first write request for a target file submitted by a user, and under the condition that the target file is not locked, a metadata server stores a corresponding relation among the ID of the file stream client, a first locking serial number and the ID of the target file in a persistent storage space. If the metadata server side breaks down (or crashes), the metadata server side does not lock the target file, and the file stream client side does not receive a locking response which is sent by the metadata server side and is used for indicating that the target file is locked, so that the file stream client side does not store information which is used for indicating that the target file is locked. In this way, the file stream client and other normal metadata servers can be consistent in cognition of whether the target file is locked or not, and deadlock is avoided.

The invention provides an object storage processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining configuration information of a target deployment region, the configuration information comprising metadata service configuration information and data storage service configuration information; corresponding metadata clients are initialized according to the metadata service configuration information, the metadata clients comprise at least two types, and each type is matched with different distributed database services; according to the data storage service configuration information, a corresponding storage rear end is initialized, the storage rear end comprises at least two types, and each type is adaptive to different object storage services; performing service registration on the initialized metadata client and the storage rear end to form an executable object storage service instance; and generating a uniform object storage service interface based on the registered object storage service instance.

A system in which an event manager executes a multitenant computing platform. The event manager obtains a first domain-specific event indicating a user request to perform operations associated with a particular tenant's domain object. Information for the domain object associated with the domain-specific event is retrieved from a tenant database, the information including a domain object definition and instructions defining its behaviors. A protected event execution environment (sandbox) is generated on the multitenant computing platform implementing restrictions on execution of the domain object's behavior instructions. The restrictions are specific to the combination of the user and the domain object. Execution of the instructions is initiated in the protected event execution environment. A second domain-specific event which indicates a request to modify the domain object information is then obtained, and the modification of the domain object information is initiated in the protected event execution environment.

A computer-implemented method of caching multi-session data communications in a computer network, includes the steps of: (a) receiving, intercepting, or monitoring one or more data sessions between a client executing a multi-session application for retrieving a desired content object and one or more metadata services, said client communicating with the one or more metadata services to discover metadata for the content object; (b) analyzing queries and responses exchanged between the client and the one of more metadata services to discover metadata for the content object; (c) receiving or intercepting subsequent data sessions between the client and content sources; (d) identifying a data protocol used by the client and identifying data queries within the data sessions; (e) identifying the content object or portions thereof requested by the client in the data queries; and (f) determining if the content object or portions thereof are stored in cache and, if so, sending the content object or portions thereof stored in cache to the client, and, if not, sending the data queries to the content sources, storing data responses from the content sources, and sending the data responses to the client.

The invention discloses a distributed block storage-oriented metadata request concurrency control method and system, and the method comprises the steps: firstly, carrying out the grouping of metadata according to the attribute information and business logic information of the metadata of a distributed block storage system; a metadata structure tree is established based on business logic information between root nodes of the sub-level tree structures obtained through grouping, then a tree-shaped lock mechanism is constructed based on the metadata structure tree, and fine-grained lock management is achieved; the method comprises the following steps: creating a transaction ID for a request for a time point and an RPC request type of each user RPC request, analyzing the user RPC request corresponding to each transaction ID, determining a specific node required to be processed by the request, then starting from a general control node of a tree lock, traversing all related root node paths downwards, and carrying out locking operation step by step. The availability of the metadata service is ensured, the requests on different sub-trees are allowed to be executed concurrently due to the fine-grained characteristic of the tree lock, and the concurrency and throughput of the metadata service are improved.

The embodiment of the invention discloses a metadata management method and related equipment, belongs to the technical field of distributed file systems, and is used for balancing metadata management capacity and concurrent access performance and improving metadata management efficiency. The method comprises a management node applied to a distributed file system, the distributed storage system further comprises a metadata node, and the metadata node is used for storing metadata. The method comprises the following steps: acquiring path feature information from a data warehouse metadata service, wherein the path feature information is used for representing a directory hierarchical relationship followed by a metadata node when the metadata node stores metadata; determining routing node information based on the path feature information; sending the routing node information to the client; the routing node information is used for the client to determine a target metadata node corresponding to the access request, and the access request is used for performing access operation on the target metadata node.

Novel techniques are disclosed for providing vPLC-specific metadata service including customized vPLC-specific metadata. In certain embodiments, each vPLC may generate a customized metadata using its corresponding vPLC-specific customization instructions. In some embodiments, a vPLC-specific metadata service may be performed using pre-generated customized vPLC-specific metadata, on-the-fly customized metadata, pre-generated CSP-format metadata, or combinations thereof.

The invention provides a metadata management method which is applied to a data system, the data system comprises a plurality of metadata service nodes and a metadata management node, each metadata service node is used for managing at least one metadata partition, and each metadata partition corresponds to part of metadata of data in the data system. The method comprises the steps that a metadata management node obtains a first query request, and a target metadata service node where a metadata partition corresponding to target data is located is determined; the metadata management node sends a second query request to the target metadata service node; and the first metadata service node in the target metadata service node obtains metadata corresponding to the target data in the metadata partition managed by the first metadata service node, and obtains the target data according to the metadata. An existing metadata storage and management method in a data system is low in efficiency, the method is high in metadata file access efficiency, and the load stability of the data system is good.

The invention relates to the technical field of distributed file systems, and discloses a metadata migration method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining the load information of each metadata service, and determining a to-be-migrated metadata service based on the load information, the to-be-migrated metadata service comprising metadata of a plurality of sub-volumes; determining metadata of the to-be-migrated sub-volume based on the popularity information of the sub-volume in the to-be-migrated metadata service; determining a plurality of clients of tenants bound with the to-be-migrated sub-volumes based on the metadata of the to-be-migrated sub-volumes; to-be-recycled metadata matched with the to-be-migrated sub-volumes are screened out from the multiple clients; releasing the to-be-recycled metadata in the metadata of the to-be-migrated sub-volume to obtain target metadata of the to-be-migrated sub-volume; and migrating the target metadata of the to-be-migrated sub-volume to the target metadata service. According to the invention, the metadata migration time of the sub-volume and the waiting time for the client to access the sub-volume are reduced.

Embodiments of the present disclosure provide techniques for implementing a service provider to collect quantum service metadata about multiple quantum systems on multiple quantum machines and storing this quantum service metadata in a service registry. The service provider may transmit a request for access with a qubit registry to each of a plurality of quantum machines, where the plurality of qubit registries store quantum service metadata associated with a plurality of quantum services configured on the plurality of quantum machines. The service provider may receive acceptance for access to each of the plurality of qubit registries. In addition, the service provider may periodically request the quantum service metadata from the plurality of qubit registries and store the received quantum service metadata in a service provider registry.

A metadata synchronization system enables real-time interactive data exploration through a distributed metadata architecture. The system propagates metadata updates through an event-based synchronization path that maintains metadata consistency across system components. For discovered data sources, the system concurrently manages metadata in primary and secondary services instead of following traditional batch synchronization. A primary metadata service generates and manages metadata definitions while an event bus component propagates metadata update events to a secondary service maintaining a local metadata store. A query service provides immediate access to metadata for data exploration operations. In some implementations, the event bus components enables near-rime metadata availability. In some implementations, the secondary metadata service processes direct metadata updates and maintains metadata states prior to synchronization with the primary metadata service. The system reduces metadata access latency by eliminating batch synchronization overhead, enables immediate data exploration through coordinated metadata management, and maintains consistency through stateful task tracking.

A hybrid transactional / analytical processing (HTAP) database is provided, which includes an online transaction processing (OLTP) engine, an online analytical processing (OLAP) engine provided separately from the OLTP engine, and a shared storage holding data. The shared storage includes an OLTP storage holding data in row format for responding to queries of the OLTP engine, and an OLAP storage holding data in column format for responding to queries of the OLAP engine. The OLAP storage includes a delta store configured to record changes made to the OLTP storage in an update, and a base store configured to implement the update from the delta store on data in persistent storage in a flush. The HTAP database further includes a table metadata service configured to generate and perform operations on table metadata of the data that is updated in the flush.

A lakehouse metadata tag creation method, a lakehouse metadata tag query method and an electronic device are provided. The method includes: after a metadata service gateway determines a metadata tag creation request based on a metadata tag creation parameter provided by a data engine, sending, by the metadata service gateway, the metadata tag creation request to a metadata tag service system, so that the metadata tag service system executes a tag creation logic based on the metadata tag creation parameter carried in the metadata tag creation request, to implement a requirement of tagging one or more pieces of metadata.

The invention discloses a read-write method and device of a disk file, electronic equipment and a storage medium, and relates to the technical field of computers.The method comprises the steps that under the condition that a forwarding mark of a to-be-read-write file is stored in latest metadata information of the to-be-read-write file, a read-write forwarding message is constructed; the forwarding mark is cached to the latest metadata information under the condition that a plurality of clients simultaneously initiate metadata opening requests for the to-be-read-write file, and the forwarding mark represents that the read-write request for the to-be-read-write file is forwarded to the target centralized control client; the target centralized control client is a client which uniformly processes concurrent read-write requests of a plurality of clients for the file to be read and written; and forwarding the read-write forwarding message to the target centralized control client so as to perform read-write operation on the file to be read and written in the disk through the target centralized control client. Through the method and the device, the problems of slow system response time and small throughput caused by large interaction pressure between metadata service and multiple clients in the related technology are solved.

The invention relates to a cloud database computing cache system and an implementation method, and the system comprises a decentralized cache architecture module which is used for eliminating a centralized bottleneck, supporting ten-thousand-level concurrency and adapting to a big and small file mixed scene; the memory object sharing module is used for reducing intermediate result processing delay through intermediate result full-life-cycle zero-copy circulation and dynamic storage decision based on a memory pool and metadata service of the memory object sharing module; the self-adaptive asynchronous write-in module is used for connecting remote storage, multiplexing memory object sharing module resources and combining mechanisms such as delay confirmation, so that the write response speed is increased, remote requests are reduced, data integrity is guaranteed, and meanwhile, persistent support is provided for intermediate results. Through cooperative operation of the three modules, the cloud database computing cache system and the implementation method can effectively adapt to the requirements of high concurrency, large and small file mixing and elastic capacity expansion and contraction of a cloud native OLAP scene, and the integrity of data transmission and the stability of system operation are guaranteed.

The invention relates to the technical field of distributed file storage, and provides a metadata service management method and device, electronic equipment and a storage medium, and the method comprises the steps: determining a target server from a server set when the current activation number of metadata services in a cluster is smaller than a target activation number; carrying out availability check on each metadata service which is currently in a backup state on the target server, if available metadata services exist, carrying out activation, and updating the current activation quantity; and if the updated current activation number is smaller than the target activation number, continuing to determine a new target server from the server set until the updated current activation number is equal to the target activation number. According to the method, the server with the largest backup number of the currently owned metadata services is selected as the target server every time, and the available metadata services on the target server are activated, so that the load balance among the servers can be realized, and the metadata service capability with higher performance is provided.

The present application discloses a disk file reading and writing method and device, electronic device and storage medium, and relates to the field of computer technology. The method includes: constructing a read-write forwarding message when a forwarding mark of a file to be read and written is stored in the latest metadata information of the file to be read and written; the forwarding mark is cached in the latest metadata information when multiple clients simultaneously initiate metadata open requests for the file to be read and written, and the forwarding mark indicates that the read and write requests for the file to be read and written are forwarded to a target centralized control client; the target centralized control client refers to a client that uniformly processes concurrent read and write requests from multiple clients for the file to be read and written; forwarding the read and write forwarding message to the target centralized control client, so that the target centralized control client can perform read and write operations on the file to be read and written on the disk. This application solves the problem in the related art that the interaction pressure between the metadata service and multiple clients is high, resulting in slow system response time and low throughput.

A lakehouse metadata tag creation method, a lakehouse metadata tag query method and an electronic device are provided. The method includes: after a metadata service gateway determines a metadata tag creation request based on a metadata tag creation parameter provided by a data engine, sending, by the metadata service gateway, the metadata tag creation request to a metadata tag service system, so that the metadata tag service system executes a tag creation logic based on the metadata tag creation parameter carried in the metadata tag creation request, to implement a requirement of tagging one or more pieces of metadata.

A method of generating a lakehouse metadata service log, a method of querying a lakehouse metadata service log, an electrode device and a storage medium are provided. The method of generating a lakehouse metadata service log includes: after a metadata server receives a data processing request sent by a data engine, determining, by the metadata server, first request identification information based on the data processing request, such that the first request identification information is used for identifying the data processing request; and then executing, by the metadata server, the request processing logic corresponding to the data processing request, and generating a log corresponding to the request processing logic based on the first request identification information, such that the log carries the first request identification information.

The invention discloses a file quota management method and device, electronic equipment and a readable storage medium, and is applied to a client, the client is in communication connection with a quota server and a metadata server, and the method comprises the following steps: determining an original file and a target file corresponding to a file change operation, sending a metadata request to a metadata server to which the original file and the target file belong; receiving an original file size and a target file size fed back by the metadata server based on the metadata request; according to the local cache file information, the size of the original file and the size of the target file, quota variation of the original file and the target file is obtained through calculation; and sending the quota variation to a quota server, so that the quota server performs capacity quota management on the original file and the target file according to the quota variation. By applying the technical scheme provided by the invention, more accurate quota management can be carried out on the file before and after the change in a file change scene.