49 results about "Metadata service" patented technology

The invention provides a heterogeneous database query method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out security verification processing on a plurality of SQL execution requests sent by a user side, and carrying out SPI mechanism scanning processing on a plurality of database plug-ins in a heterogeneous database plug-in directory based on a high-performance database connection pool after the security verification processing is passed, determining a target database plug-in package corresponding to each SQL execution request; and performing metadata query processing on the SQL execution request in the target data source based on a database operation client interface and a metadata service interface under the target database plug-in package, and determining a structured tree result. According to the method, an extensible and dynamically integrated heterogeneous database plug-in mechanism is realized, unified access of various databases is supported, the platform maintenance cost is remarkably reduced, hierarchical metadata of any type of databases is obtained through a consistent metadata service interface, and the user experience and the cross-database query efficiency are improved.

The invention discloses a distributed block storage-oriented metadata request concurrency control method and system, and the method comprises the steps: firstly, carrying out the grouping of metadata according to the attribute information and business logic information of the metadata of a distributed block storage system; a metadata structure tree is established based on business logic information between root nodes of the sub-level tree structures obtained through grouping, then a tree-shaped lock mechanism is constructed based on the metadata structure tree, and fine-grained lock management is achieved; the method comprises the following steps: creating a transaction ID for a request for a time point and an RPC request type of each user RPC request, analyzing the user RPC request corresponding to each transaction ID, determining a specific node required to be processed by the request, then starting from a general control node of a tree lock, traversing all related root node paths downwards, and carrying out locking operation step by step. The availability of the metadata service is ensured, the requests on different sub-trees are allowed to be executed concurrently due to the fine-grained characteristic of the tree lock, and the concurrency and throughput of the metadata service are improved.

Novel techniques are disclosed for providing vPLC-specific metadata service including customized vPLC-specific metadata. In certain embodiments, each vPLC may generate a customized metadata using its corresponding vPLC-specific customization instructions. In some embodiments, a vPLC-specific metadata service may be performed using pre-generated customized vPLC-specific metadata, on-the-fly customized metadata, pre-generated CSP-format metadata, or combinations thereof.

Embodiments of the present disclosure provide techniques for implementing a service provider to collect quantum service metadata about multiple quantum systems on multiple quantum machines and storing this quantum service metadata in a service registry. The service provider may transmit a request for access with a qubit registry to each of a plurality of quantum machines, where the plurality of qubit registries store quantum service metadata associated with a plurality of quantum services configured on the plurality of quantum machines. The service provider may receive acceptance for access to each of the plurality of qubit registries. In addition, the service provider may periodically request the quantum service metadata from the plurality of qubit registries and store the received quantum service metadata in a service provider registry.

A metadata synchronization system enables real-time interactive data exploration through a distributed metadata architecture. The system propagates metadata updates through an event-based synchronization path that maintains metadata consistency across system components. For discovered data sources, the system concurrently manages metadata in primary and secondary services instead of following traditional batch synchronization. A primary metadata service generates and manages metadata definitions while an event bus component propagates metadata update events to a secondary service maintaining a local metadata store. A query service provides immediate access to metadata for data exploration operations. In some implementations, the event bus components enables near-rime metadata availability. In some implementations, the secondary metadata service processes direct metadata updates and maintains metadata states prior to synchronization with the primary metadata service. The system reduces metadata access latency by eliminating batch synchronization overhead, enables immediate data exploration through coordinated metadata management, and maintains consistency through stateful task tracking.

A hybrid transactional / analytical processing (HTAP) database is provided, which includes an online transaction processing (OLTP) engine, an online analytical processing (OLAP) engine provided separately from the OLTP engine, and a shared storage holding data. The shared storage includes an OLTP storage holding data in row format for responding to queries of the OLTP engine, and an OLAP storage holding data in column format for responding to queries of the OLAP engine. The OLAP storage includes a delta store configured to record changes made to the OLTP storage in an update, and a base store configured to implement the update from the delta store on data in persistent storage in a flush. The HTAP database further includes a table metadata service configured to generate and perform operations on table metadata of the data that is updated in the flush.

The invention relates to a cloud database computing cache system and an implementation method, and the system comprises a decentralized cache architecture module which is used for eliminating a centralized bottleneck, supporting ten-thousand-level concurrency and adapting to a big and small file mixed scene; the memory object sharing module is used for reducing intermediate result processing delay through intermediate result full-life-cycle zero-copy circulation and dynamic storage decision based on a memory pool and metadata service of the memory object sharing module; the self-adaptive asynchronous write-in module is used for connecting remote storage, multiplexing memory object sharing module resources and combining mechanisms such as delay confirmation, so that the write response speed is increased, remote requests are reduced, data integrity is guaranteed, and meanwhile, persistent support is provided for intermediate results. Through cooperative operation of the three modules, the cloud database computing cache system and the implementation method can effectively adapt to the requirements of high concurrency, large and small file mixing and elastic capacity expansion and contraction of a cloud native OLAP scene, and the integrity of data transmission and the stability of system operation are guaranteed.

This application provides a method and system for repairing temporary check blocks based on erasure coding in distributed block storage. The method includes: when the distributed block storage system detects a check block write failure, a metadata service creates a temporary check block according to the erasure coding pattern; the temporary check block records operation logs and performs data writing, replaying the IO operation according to the data version number; the operation logs of the temporary check block are used to repair data block inconsistencies; and the check block data is quickly restored using the data bitmap of the temporary check block. This method, through its rapid recovery processing of temporary check blocks, can directly restore check block data without encoding, reducing internal IO and accelerating the recovery speed of erasure-coded check blocks.

The invention relates to generation of table metadata in an HTAP database during flushing. A hybrid transactional / analytic processing (HTAP) database is provided that includes an online transactional processing (OLTP) engine, an online analytic processing (OLAP) engine disposed separately from the OLTP engine, and a shared storage that holds data. The shared storage device includes an OLTP storage device that holds data for responding to a query of the OLTP engine in a row format, and an OLAP storage device that holds data for responding to a query of the OLAP engine in a column format. The OLAP storage device includes: an incremental storage region configured to record a change made to the OLTP storage device at the time of update; and a base storage area configured to implement updates from the incremental storage area on the data in the persistent storage device during flushing. The HTAP database also includes a table metadata service configured to generate table metadata of the data that is updated at the time of flushing and perform an operation on the table metadata of the data that is updated at the time of flushing.

Techniques are provided for maintaining and utilizing a file index and a file version index. Metadata may be evaluated to identify constant attributes and modifiable attributes of files. A file index of a file catalog may be populated with the constant attributes. A file version index of the file catalog may be populated with the modifiable attributes as file versions of the files. In response to receiving a request for a file, the file index and the file version index are evaluated to identify a location of the file within a data source. Access to the file at the location within the data source is provided.

The invention relates to a data processing method of a data lake, computer equipment, a storage medium and a product, and relates to the technical field of big data. The method comprises the steps that a data file is written into a data lake through a first calculation instance, and metadata information of the data file is generated through the first calculation instance; the first calculation instance is any one of at least two calculation instances contained in a distributed cluster; carrying out persistent processing on metadata information of the data file through the shared storage component and storing the metadata information; the shared storage component is used for providing services for at least two computing instances. By means of the method, the risk resistance of system data can be improved, even if the metadata service is different or needs to be restarted, due to the fact that the metadata information is subjected to persistent processing, the data cannot be lost, and the operation of entering a lake of the data file can be normally conducted, the stability of the system is improved, and the user experience is improved. And the timeliness of data processing is ensured.

The application provides a Lustre file system write processing method and device, the method comprising: in response to a write data request initiated by an application layer for a target file, detecting whether an accelerated write condition is met; if the detection result is yes, copying to-be-written data to a preset cache area in a kernel state of a Lustre client and returning write success to the application layer, applying a distributed write lock corresponding to a file offset range of the to-be-written data to a metadata service by using a background thread, and after holding the lock, writing the to-be-written data in batches into a page cache; if the detection result is no, writing the to-be-written data into the page cache page by page, including applying a distributed write lock corresponding to a file offset range of the page data to the metadata service by using a background thread, and after holding the lock, writing the page data into the page cache; and persistently storing data in the page cache to an object storage server. Compared with related technologies, the application avoids multiple lock applications and improves data write efficiency.

This application discloses a database object processing method, system, device, and medium, applied in the field of database technology. In the mapping phase, the hierarchical identifiers (after removing hierarchical separators) of the original identifier key are concatenated into a hierarchical identifier string, generating a first digital fingerprint. A mapping identifier key is generated based on the target start character, the target digital fingerprint algorithm identifier, the first digital fingerprint, the target hierarchical identifier length, and the hierarchical identifier string. In the inverse mapping phase, after determining that the mapping identifier key starts with the target start character, the hierarchical identifier string is segmented based on the target hierarchical identifier length to obtain each hierarchical identifier, generating a second digital fingerprint. When the second digital fingerprint matches the first digital fingerprint, the hierarchical separator is used to restore each hierarchical identifier to the original identifier key. By constructing a unified bidirectional mapping mechanism, the database cluster can be compatible with different metadata services, significantly improving system scalability.

The application discloses a kind of metadata request concurrent control method and system for distributed block storage, first according to the attribute information and service logic information of distributed block storage system metadata, metadata is grouped, and the root node between the sub-level tree structure based on grouping is obtained Service logic information establishes to obtain metadata structure tree, then based on metadata structure tree, tree-shaped lock mechanism is constructed, and fine-grained lock management is realized;For the time point and RPC request type of each user RPC request, transaction ID is created for request, by analyzing the user RPC request corresponding to each transaction ID, the specific node that request needs to process is determined, then from the total control node of tree-shaped lock, all relevant root node paths are traversed downwards, and locking operation is performed step by step, to ensure the availability of metadata service, the fine-grained characteristics of tree-shaped lock allow concurrent execution of requests on different sub-trees, improve the concurrency and throughput of metadata service.

The invention provides a metadata control method, device and equipment based on a separated architecture file system, and relates to the technical field of metadata processing. A metadata control method based on a separate architecture file system is applied to a client and comprises the following steps: acquiring a user request to determine target metadata of the file system and a metadata operation type corresponding to the target metadata; decomposing a server side path corresponding to the user request to determine the position of each metadata corresponding to the user request; and based on the position of each metadata corresponding to the user request, respectively carrying out operations corresponding to the metadata operation type through a preset remote direct memory access operation. According to the embodiment of the invention, the overall throughput capacity of the metadata service of the file system can be improved.

This invention discloses a meteorological data external service process and system based on DMZ secure isolation. The process includes the synchronous acquisition of shared meteorological data, meteorological data processing, and external service. A DMZ zone is introduced between the meteorological intranet and the Internet as an isolation zone for data transmission. Then, data element extraction and regional clipping are performed. The system includes a hardware support unit, a software support unit, a data service unit, a business support unit, a business application unit, and a user unit. The hardware support unit ensures the security of the intranet and DMZ through a tiered, one-way access policy via firewall. The software support unit supports the operation and service of upper-layer applications. The data service unit is responsible for acquiring meteorological data and industry-specific data. The business support unit provides business support. The business application unit provides meteorological data services and sharing functions to users. This invention enables secure transmission and efficient sharing of meteorological data for external service systems.

The invention relates to the technical field of data management, and discloses a general data management method and system for engineering construction, and the system comprises a meta-model management module, a data dictionary management module, a code table management module, a metadata instantiation module, and a metadata service module. According to the method, based on an aggregation structure of a core metadata model, identification, quality, content, distribution, materials and equipment are forcibly included, the method is responsible for six core sub-models of a unit, and standardized description of engineering construction materials and equipment data is achieved; metadata entities and elements are defined through data dictionary specifications, and semantic ambiguity is reduced; an enumeration value domain is managed in a unified mode in combination with a standardized code table, and it is guaranteed that data are unambiguous; and dynamically adapting to new materials, new attributes and new business scene requirements based on a metadata expansion mechanism. According to the invention, the machine readability and operability of data are improved, and the flexibility and long-term applicability of data management in the full life cycle of a project are improved.

Methods, systems, and devices for data management are described. A first application in a destination computing environment of a data management system (DMS) may determine that a bulk-push criterion is satisfied for a second application in a source computing environment of the DMS. The first application may transmit, to an asynchronous metadata service, a request indicating the second application for which the bulk-push criterion is satisfied. The request may be configured to cause the asynchronous metadata service to query a database in the source computing environment, identify a latest version of one or more rows that include metadata associated with the second application, and generate data records indicating the latest version of the one or more rows that include the metadata associated with the second application. The first application may receive the data records via an asynchronous data stream between the first application and the second application.

The invention provides a cache management method and device for metadata of a file system based on a separated architecture, and relates to the technical field of metadata processing. The metadata cache management method of the file system based on the separated architecture is applied to a client and comprises the following steps: configuring a metadata lease corresponding to the client; based on the metadata lease, metadata caching operation of the file system is carried out through preset remote direct memory access operation; and updating the metadata in the local cache of the client based on the metadata lease according to the result of the metadata cache operation. According to the embodiment of the invention, the execution delay of the metadata query can be reduced, and the overall throughput capacity of the metadata service is improved.

The invention discloses a rapid rendering system and method for meteorological grid data, and the method comprises the steps: (1) carrying out the preprocessing of the meteorological grid data, and converting the meteorological grid data into the meteorological grid data which is suitable for the rendering of a three-dimensional space and is of a unified data structure; (2) performing spatial optimization on the meteorological grid data processed in the step (1) by utilizing a view cone cutting technology; and (3) in the visualization process, only loading and rendering the meteorological grid data in the view cone for visualization, and dynamically updating the visualized meteorological grid data. The system comprises a data processing unit, a data service unit, a visualization unit and a user interaction unit. The data processing unit is in communication connection with the data service unit, the data service unit is in communication connection with the visualization unit, and the data service unit and the visualization unit are respectively in communication connection with the user interaction unit. According to the invention, the fluency and interaction experience of meteorological grid data visualization can be improved.

The application provides a distributed file storage metadata service method, a server, a client and a system. The method comprises the following steps: the server receives a request by using a user mode protocol stack and converts the request into a KV instruction to execute on a KV SSD; the KV SSD maintains two types of key-value pairs: the first type of key-value pair takes a file ID generated based on file original identity information and combined with an attribute identifier as a Key, and stores metadata attribute values; the second type of key-value pair takes a file current path as a Key, and stores original identity information. When accessing, the first type of key-value pair is preferentially searched by using a candidate ID calculated based on the current path; if no hit is found, the second type of key-value pair is used to obtain the original identity information to determine a real file ID and perform an operation. The application does not need to query a directory layer by layer, can locate metadata in at most two interactions, eliminates performance fluctuation, does not need to cache a directory tree, significantly reduces memory consumption and improves scalability.

The present application relates to the technical field of distributed storage. Disclosed are a metadata addressing method and apparatus, and a medium and a product. A target number of metadata service paths are obtained by filtering metadata feature parameters within a preset time, such that local management is performed on metadata in a large-scale distributed storage file system, and path information of the metadata is recorded and counted to improve the locality management of the metadata; moreover, a first path result is obtained by means of granularity calculation, and granularity selection is performed on the metadata paths to improve the temporal locality of the metadata; in addition, a second path result is obtained by evaluating and calculating the first path result on the basis of a path optimization strategy, such that locality processing is performed on the second path result on the basis of a locality function. Thus, the locality data structure of metadata is changed, thereby ensuring the spatial locality of metadata storage paths in a distributed file system, optimizing the paths for metadata operations in the distributed storage file system and improving the I / O performance of a storage system.

A hybrid transactional / analytical processing (HTAP) database is provided, which includes an online transaction processing (OLTP) engine, an online analytical processing (OLAP) engine provided separately from the OLTP engine, and a shared storage holding data. The shared storage includes an OLTP storage holding data in row format for responding to queries of the OLTP engine, and an OLAP storage holding data in column format for responding to queries of the OLAP engine. The OLAP storage includes a delta store configured to record changes made to the OLTP storage in an update, and a base store configured to implement the update from the delta store on data in persistent storage in a flush. The HTAP database further includes a table metadata service configured to generate and perform operations on table metadata of the data that is updated in the flush.

Systems and processes are described for a message service with distributed key caching for server-side encryption. Message requests are received by message handlers of the message service that cache data encryption keys used to encrypt and decrypt messages that are stored to message containers in back end storage. A metadata service obtains the data encryption keys from a key management service, caches the keys locally, and sends the keys to the message handlers upon request, where the keys are cached, again. The key management service may generate the data encryption keys based on a master key (e.g., a client's master key). The message handlers may send both message data encrypted using the data encryption key and an encrypted copy of the data encryption key to be stored together in the data store.

This application provides a method and system for processing multi-source heterogeneous data, an electronic device, and a storage medium. The method includes: a data access layer initiating a mode request to a metadata service before sending a message to a message bus via a target component, wherein the target component is one of the following: an adapter or a processing component; the metadata service responding to the mode request sends constraint information corresponding to the target component to the target component, wherein the constraint information includes: field rules and type constraints; the target component verifies the acquired target source data according to the constraint information, and if the target source data satisfies the constraint information, pushes the target source data to the message bus; the message bus persistently stores the target source data. This application solves the technical problem of poor overall system stability and reliability in related technologies.

The invention relates to an exception handling method and device, equipment and a storage medium. The method comprises the steps of determining an abnormal type of an abnormal client according to a network operation condition of the abnormal client and an access authorization condition to a cluster node when detecting that the abnormal client exists in clients associated with the cluster node, and sending the abnormal client to the cluster node when the abnormal type is an operation abnormal type. And according to the running state of the metadata service in the abnormal client and a local log file corresponding to the abnormal client, determining an anomaly analysis result of the abnormal client, the local log file being generated by the cluster node by analyzing a data anomaly phenomenon existing in the abnormal client. By adopting the method, the accuracy of anomaly analysis can be improved.

A computer-implemented method can include receiving a proposal to modify data stored in a distributed and replicated file system coupled to a network, the distributed and replicated data system including a plurality of nodes, each node including a server; a metadata service maintaining and updating a copy of a namespace of the distributed and replicated file system and coordinating updates to the data by generating an ordered set of protocols corresponding to the received proposal, the ordered set of protocols specifying an order of nodes to modify data stored in the data nodes and cause corresponding changes in a state of the namespace. For each protocol in the generated ordered set of protocols, a corresponding writer list can be provided, the corresponding writer list including an ordered list of nodes to execute the protocol and make the corresponding changes to the namespace. The ordered set of protocols can then be sent to the plurality of nodes along with the corresponding writer list or a pre-generated index thereof for each protocol in the ordered set of protocols, and each node in the plurality of nodes can be configured to only execute the protocol of the first listed node on the received writer list.

The application discloses a kind of distributed file systems, comprising: metadata service module, data storage service module, garbage collection service module and access service module, access service module finds first file number and corresponding each first logic unit according to the write request sent by client, the first data slot is obtained by hash calculation to first file number and the number of each first logic unit, and write request is distributed to the target data storage service module corresponding to first data slot, so that the target data storage service module will write the metadata and real data of write request to the latest data file of the underlying storage system responsible for first data slot, update the file data index cached in first data slot after writing success.The application all upper write request will be converted into the additional write of underlying storage system data file, support the random write of business file, and the complexity is low, and upper logic does not need to care about the migration of underlying data, and the maintenance difficulty is low.

This disclosure describes a technique for providing data consistency for hybrid transactional and analytical processing. A logical log and a Log Sequence Number (LSN) associated with the logical log can be generated based on data captured by a first processing engine. The logical log and LSN can be propagated to a storage subsystem configured to communicate with both the first and second processing engines. The LSN and information indicating the LSN mode version can be stored and distributed by a metadata service. The first processing engine, the second processing engine, the storage subsystem, and the metadata service are modularized and support an LSN mechanism for maintaining data consistency.

The application relates to the technical field of distributed storage, in particular to a file processing method of a distributed cluster, a distributed cluster and a computing device. The distributed cluster comprises a computing device, a storage device, a first metadata service and a second metadata service. The first metadata service is configured on the computing device, the second metadata service is configured on the storage device or other devices except the computing device and the storage device, and a file directory corresponding to the second metadata service comprises a file directory corresponding to the first metadata service. The method comprises the following steps: the computing device receives a file reading task, the file reading task comprises a file directory of a first file to be read, and the first file is located on the storage device; based on the first metadata service configured on the computing device, first metadata corresponding to the file directory of the first file and managed by the second metadata service is acquired; and the file reading task of the first file in the storage device is executed according to the first metadata.