73 results about "Metadata service" patented technology

Novel techniques are disclosed for providing vPLC-specific metadata service including customized vPLC-specific metadata. In certain embodiments, each vPLC may generate a customized metadata using its corresponding vPLC-specific customization instructions. In some embodiments, a vPLC-specific metadata service may be performed using pre-generated customized vPLC-specific metadata, on-the-fly customized metadata, pre-generated CSP-format metadata, or combinations thereof.

The invention provides a temporary check block repairing method and system based on erasure codes in distributed block storage, and the method comprises the steps that a distributed block storage system detects that a check block fails to be written in, and a metadata service creates a temporary check block according to an erasure code mode; the temporary check block records an operation log and executes data writing, and replays IO operation according to a data version number; repairing the inconsistency of the data blocks by using the operation log of the temporary check block; and quickly recovering the check block data through the data bitmap of the temporary check block. According to the rapid recovery processing method for the temporary check block, the check block data can be directly recovered without a coding mode, the internal IO is reduced, and the recovery speed of the erasure code check block is increased.

The invention provides a heterogeneous database query method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out security verification processing on a plurality of SQL execution requests sent by a user side, and carrying out SPI mechanism scanning processing on a plurality of database plug-ins in a heterogeneous database plug-in directory based on a high-performance database connection pool after the security verification processing is passed, determining a target database plug-in package corresponding to each SQL execution request; and performing metadata query processing on the SQL execution request in the target data source based on a database operation client interface and a metadata service interface under the target database plug-in package, and determining a structured tree result. According to the method, an extensible and dynamically integrated heterogeneous database plug-in mechanism is realized, unified access of various databases is supported, the platform maintenance cost is remarkably reduced, hierarchical metadata of any type of databases is obtained through a consistent metadata service interface, and the user experience and the cross-database query efficiency are improved.

The invention provides an object storage processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining configuration information of a target deployment region, the configuration information comprising metadata service configuration information and data storage service configuration information; corresponding metadata clients are initialized according to the metadata service configuration information, the metadata clients comprise at least two types, and each type is matched with different distributed database services; according to the data storage service configuration information, a corresponding storage rear end is initialized, the storage rear end comprises at least two types, and each type is adaptive to different object storage services; performing service registration on the initialized metadata client and the storage rear end to form an executable object storage service instance; and generating a uniform object storage service interface based on the registered object storage service instance.

A system in which an event manager executes a multitenant computing platform. The event manager obtains a first domain-specific event indicating a user request to perform operations associated with a particular tenant's domain object. Information for the domain object associated with the domain-specific event is retrieved from a tenant database, the information including a domain object definition and instructions defining its behaviors. A protected event execution environment (sandbox) is generated on the multitenant computing platform implementing restrictions on execution of the domain object's behavior instructions. The restrictions are specific to the combination of the user and the domain object. Execution of the instructions is initiated in the protected event execution environment. A second domain-specific event which indicates a request to modify the domain object information is then obtained, and the modification of the domain object information is initiated in the protected event execution environment.

The invention discloses a distributed block storage-oriented metadata request concurrency control method and system, and the method comprises the steps: firstly, carrying out the grouping of metadata according to the attribute information and business logic information of the metadata of a distributed block storage system; a metadata structure tree is established based on business logic information between root nodes of the sub-level tree structures obtained through grouping, then a tree-shaped lock mechanism is constructed based on the metadata structure tree, and fine-grained lock management is achieved; the method comprises the following steps: creating a transaction ID for a request for a time point and an RPC request type of each user RPC request, analyzing the user RPC request corresponding to each transaction ID, determining a specific node required to be processed by the request, then starting from a general control node of a tree lock, traversing all related root node paths downwards, and carrying out locking operation step by step. The availability of the metadata service is ensured, the requests on different sub-trees are allowed to be executed concurrently due to the fine-grained characteristic of the tree lock, and the concurrency and throughput of the metadata service are improved.

The embodiment of the invention discloses a metadata management method and related equipment, belongs to the technical field of distributed file systems, and is used for balancing metadata management capacity and concurrent access performance and improving metadata management efficiency. The method comprises a management node applied to a distributed file system, the distributed storage system further comprises a metadata node, and the metadata node is used for storing metadata. The method comprises the following steps: acquiring path feature information from a data warehouse metadata service, wherein the path feature information is used for representing a directory hierarchical relationship followed by a metadata node when the metadata node stores metadata; determining routing node information based on the path feature information; sending the routing node information to the client; the routing node information is used for the client to determine a target metadata node corresponding to the access request, and the access request is used for performing access operation on the target metadata node.

Novel techniques are disclosed for providing vPLC-specific metadata service including customized vPLC-specific metadata. In certain embodiments, each vPLC may generate a customized metadata using its corresponding vPLC-specific customization instructions. In some embodiments, a vPLC-specific metadata service may be performed using pre-generated customized vPLC-specific metadata, on-the-fly customized metadata, pre-generated CSP-format metadata, or combinations thereof.

Embodiments of the present disclosure provide techniques for implementing a service provider to collect quantum service metadata about multiple quantum systems on multiple quantum machines and storing this quantum service metadata in a service registry. The service provider may transmit a request for access with a qubit registry to each of a plurality of quantum machines, where the plurality of qubit registries store quantum service metadata associated with a plurality of quantum services configured on the plurality of quantum machines. The service provider may receive acceptance for access to each of the plurality of qubit registries. In addition, the service provider may periodically request the quantum service metadata from the plurality of qubit registries and store the received quantum service metadata in a service provider registry.

A metadata synchronization system enables real-time interactive data exploration through a distributed metadata architecture. The system propagates metadata updates through an event-based synchronization path that maintains metadata consistency across system components. For discovered data sources, the system concurrently manages metadata in primary and secondary services instead of following traditional batch synchronization. A primary metadata service generates and manages metadata definitions while an event bus component propagates metadata update events to a secondary service maintaining a local metadata store. A query service provides immediate access to metadata for data exploration operations. In some implementations, the event bus components enables near-rime metadata availability. In some implementations, the secondary metadata service processes direct metadata updates and maintains metadata states prior to synchronization with the primary metadata service. The system reduces metadata access latency by eliminating batch synchronization overhead, enables immediate data exploration through coordinated metadata management, and maintains consistency through stateful task tracking.

A hybrid transactional / analytical processing (HTAP) database is provided, which includes an online transaction processing (OLTP) engine, an online analytical processing (OLAP) engine provided separately from the OLTP engine, and a shared storage holding data. The shared storage includes an OLTP storage holding data in row format for responding to queries of the OLTP engine, and an OLAP storage holding data in column format for responding to queries of the OLAP engine. The OLAP storage includes a delta store configured to record changes made to the OLTP storage in an update, and a base store configured to implement the update from the delta store on data in persistent storage in a flush. The HTAP database further includes a table metadata service configured to generate and perform operations on table metadata of the data that is updated in the flush.

A lakehouse metadata tag creation method, a lakehouse metadata tag query method and an electronic device are provided. The method includes: after a metadata service gateway determines a metadata tag creation request based on a metadata tag creation parameter provided by a data engine, sending, by the metadata service gateway, the metadata tag creation request to a metadata tag service system, so that the metadata tag service system executes a tag creation logic based on the metadata tag creation parameter carried in the metadata tag creation request, to implement a requirement of tagging one or more pieces of metadata.

The invention relates to a cloud database computing cache system and an implementation method, and the system comprises a decentralized cache architecture module which is used for eliminating a centralized bottleneck, supporting ten-thousand-level concurrency and adapting to a big and small file mixed scene; the memory object sharing module is used for reducing intermediate result processing delay through intermediate result full-life-cycle zero-copy circulation and dynamic storage decision based on a memory pool and metadata service of the memory object sharing module; the self-adaptive asynchronous write-in module is used for connecting remote storage, multiplexing memory object sharing module resources and combining mechanisms such as delay confirmation, so that the write response speed is increased, remote requests are reduced, data integrity is guaranteed, and meanwhile, persistent support is provided for intermediate results. Through cooperative operation of the three modules, the cloud database computing cache system and the implementation method can effectively adapt to the requirements of high concurrency, large and small file mixing and elastic capacity expansion and contraction of a cloud native OLAP scene, and the integrity of data transmission and the stability of system operation are guaranteed.

The application discloses a load balancing method for distributed jobs, an electronic device and a storage medium. The load balancing method for distributed jobs is applied to a metadata service node and comprises the following steps: determining a trigger and a default execution node of the trigger meeting a preset condition, and acquiring resource usage of all execution nodes; determining a target execution node from all the execution nodes according to the resource usage, and establishing dynamic distribution information of the trigger and the target execution node in a preset global trigger execution chain table; and sending a trigger identifier of the trigger to the target execution node according to the dynamic distribution information, so that the target execution node executes a job task corresponding to the trigger. According to the embodiment of the application, the dynamic distribution of the execution node of the trigger is realized, the problem of long job execution time caused by the unbalanced load of the execution node is prevented, and the execution efficiency of the job task is improved.

This application provides a method and system for repairing temporary check blocks based on erasure coding in distributed block storage. The method includes: when the distributed block storage system detects a check block write failure, a metadata service creates a temporary check block according to the erasure coding pattern; the temporary check block records operation logs and performs data writing, replaying the IO operation according to the data version number; the operation logs of the temporary check block are used to repair data block inconsistencies; and the check block data is quickly restored using the data bitmap of the temporary check block. This method, through its rapid recovery processing of temporary check blocks, can directly restore check block data without encoding, reducing internal IO and accelerating the recovery speed of erasure-coded check blocks.

The invention relates to generation of table metadata in an HTAP database during flushing. A hybrid transactional / analytic processing (HTAP) database is provided that includes an online transactional processing (OLTP) engine, an online analytic processing (OLAP) engine disposed separately from the OLTP engine, and a shared storage that holds data. The shared storage device includes an OLTP storage device that holds data for responding to a query of the OLTP engine in a row format, and an OLAP storage device that holds data for responding to a query of the OLAP engine in a column format. The OLAP storage device includes: an incremental storage region configured to record a change made to the OLTP storage device at the time of update; and a base storage area configured to implement updates from the incremental storage area on the data in the persistent storage device during flushing. The HTAP database also includes a table metadata service configured to generate table metadata of the data that is updated at the time of flushing and perform an operation on the table metadata of the data that is updated at the time of flushing.

Techniques are provided for maintaining and utilizing a file index and a file version index. Metadata may be evaluated to identify constant attributes and modifiable attributes of files. A file index of a file catalog may be populated with the constant attributes. A file version index of the file catalog may be populated with the modifiable attributes as file versions of the files. In response to receiving a request for a file, the file index and the file version index are evaluated to identify a location of the file within a data source. Access to the file at the location within the data source is provided.

The invention provides a data query method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of big data. The method comprises the following steps: receiving a user query request; when target metadata corresponding to the user query request is queried in the active cache database, obtaining the target metadata from the active cache database; when the target metadata is not queried in the active cache database and the target metadata is queried in the engine database through the query engine, obtaining the target metadata from the engine database; when the target metadata is not queried in the active cache database and the target metadata is not queried in the engine database through the query engine, the query engine requests the metadata management system to pull the target metadata in the data lake; according to the embodiment of the invention, the quasi-real-time query data is provided for the business party on the premise of ensuring the query performance, the freshness of the data is ensured, and the pressure on the metadata service of the data lake is reduced.

The invention relates to a data processing method of a data lake, computer equipment, a storage medium and a product, and relates to the technical field of big data. The method comprises the steps that a data file is written into a data lake through a first calculation instance, and metadata information of the data file is generated through the first calculation instance; the first calculation instance is any one of at least two calculation instances contained in a distributed cluster; carrying out persistent processing on metadata information of the data file through the shared storage component and storing the metadata information; the shared storage component is used for providing services for at least two computing instances. By means of the method, the risk resistance of system data can be improved, even if the metadata service is different or needs to be restarted, due to the fact that the metadata information is subjected to persistent processing, the data cannot be lost, and the operation of entering a lake of the data file can be normally conducted, the stability of the system is improved, and the user experience is improved. And the timeliness of data processing is ensured.

The embodiment of the invention provides a lake and warehouse integrated data processing system, and relates to the field of big data processing. A data access layer accesses data from a plurality of heterogeneous data sources; the object storage unit is used for persistently storing the initial data accessed from the data access layer in an original data format to form an original data area; the core storage engine unit adopts a large-scale parallel processing architecture and manages a refined data area through an incremental data snapshot mechanism; the unified metadata service manages metadata in a unified manner, and records a derivative relationship and structure mapping; the data processing layer extracts data from the original data area, performs conversion and loading operation, writes the processed data into the refined data area, and is integrated with a data quality verification module; and the data service layer provides a uniform data access interface and can query and analyze data in the original data area and the refined data area. The problems of data islands, performance bottlenecks, treatment difficulty and the like in the prior art are effectively solved.

The application provides a Lustre file system write processing method and device, the method comprising: in response to a write data request initiated by an application layer for a target file, detecting whether an accelerated write condition is met; if the detection result is yes, copying to-be-written data to a preset cache area in a kernel state of a Lustre client and returning write success to the application layer, applying a distributed write lock corresponding to a file offset range of the to-be-written data to a metadata service by using a background thread, and after holding the lock, writing the to-be-written data in batches into a page cache; if the detection result is no, writing the to-be-written data into the page cache page by page, including applying a distributed write lock corresponding to a file offset range of the page data to the metadata service by using a background thread, and after holding the lock, writing the page data into the page cache; and persistently storing data in the page cache to an object storage server. Compared with related technologies, the application avoids multiple lock applications and improves data write efficiency.

This application discloses a database object processing method, system, device, and medium, applied in the field of database technology. In the mapping phase, the hierarchical identifiers (after removing hierarchical separators) of the original identifier key are concatenated into a hierarchical identifier string, generating a first digital fingerprint. A mapping identifier key is generated based on the target start character, the target digital fingerprint algorithm identifier, the first digital fingerprint, the target hierarchical identifier length, and the hierarchical identifier string. In the inverse mapping phase, after determining that the mapping identifier key starts with the target start character, the hierarchical identifier string is segmented based on the target hierarchical identifier length to obtain each hierarchical identifier, generating a second digital fingerprint. When the second digital fingerprint matches the first digital fingerprint, the hierarchical separator is used to restore each hierarchical identifier to the original identifier key. By constructing a unified bidirectional mapping mechanism, the database cluster can be compatible with different metadata services, significantly improving system scalability.

The application discloses a kind of metadata request concurrent control method and system for distributed block storage, first according to the attribute information and service logic information of distributed block storage system metadata, metadata is grouped, and the root node between the sub-level tree structure based on grouping is obtained Service logic information establishes to obtain metadata structure tree, then based on metadata structure tree, tree-shaped lock mechanism is constructed, and fine-grained lock management is realized;For the time point and RPC request type of each user RPC request, transaction ID is created for request, by analyzing the user RPC request corresponding to each transaction ID, the specific node that request needs to process is determined, then from the total control node of tree-shaped lock, all relevant root node paths are traversed downwards, and locking operation is performed step by step, to ensure the availability of metadata service, the fine-grained characteristics of tree-shaped lock allow concurrent execution of requests on different sub-trees, improve the concurrency and throughput of metadata service.

The invention provides a metadata control method, device and equipment based on a separated architecture file system, and relates to the technical field of metadata processing. A metadata control method based on a separate architecture file system is applied to a client and comprises the following steps: acquiring a user request to determine target metadata of the file system and a metadata operation type corresponding to the target metadata; decomposing a server side path corresponding to the user request to determine the position of each metadata corresponding to the user request; and based on the position of each metadata corresponding to the user request, respectively carrying out operations corresponding to the metadata operation type through a preset remote direct memory access operation. According to the embodiment of the invention, the overall throughput capacity of the metadata service of the file system can be improved.

Techniques are provided for a maintaining and utilizing a file index and a file version index. Metadata may be evaluated to identify constant attributes and modifiable attributes of files. A file index of a file catalog may be populated with the constant attributes. A file version index of the file catalog may be populated with the modifiable attributes as file versions of the files. In response to receiving a request for a file, the file index and the file version index are evaluated to identify a location of the file within a data source. Access to the file at the location within the data source is provided.

The invention provides an environment configuration method and device of a cloud host, electronic equipment and a readable medium. The method comprises the following steps: acquiring host configuration information and environment configuration information specified by a user; configuring a cloud host conforming to the host configuration information according to a cloud host configuration and an operating system specified in the host configuration information; accessing the metadata service through the cloud host to obtain an environment deployment tool corresponding to the cloud host; and installing a software package of a software version specified by a user in the cloud host through an environment deployment tool according to the software package and the software version specified in the environment configuration information so as to perform environment configuration on the cloud host. According to the method, the operation difficulty in the deployment process and the use cost of a user can be reduced, and the user experience of the cloud host is improved.

This invention discloses a meteorological data external service process and system based on DMZ secure isolation. The process includes the synchronous acquisition of shared meteorological data, meteorological data processing, and external service. A DMZ zone is introduced between the meteorological intranet and the Internet as an isolation zone for data transmission. Then, data element extraction and regional clipping are performed. The system includes a hardware support unit, a software support unit, a data service unit, a business support unit, a business application unit, and a user unit. The hardware support unit ensures the security of the intranet and DMZ through a tiered, one-way access policy via firewall. The software support unit supports the operation and service of upper-layer applications. The data service unit is responsible for acquiring meteorological data and industry-specific data. The business support unit provides business support. The business application unit provides meteorological data services and sharing functions to users. This invention enables secure transmission and efficient sharing of meteorological data for external service systems.

The invention discloses a data synchronization method and a CDN system based on an adapter mode, and the method comprises the steps: a management service module determines a source station cluster from a plurality of object storage clusters after the authentication of a data writing application is passed; the management service module sends the write-in credential of the source station cluster to a writer, so that the writer is connected with a target storage abstraction layer of the source station cluster according to the write-in credential; the target storage abstraction layer writes the target data into a source station cluster and sends a write-in completion notification to a metadata service module; the metadata service module updates metadata and sends a synchronization message to the synchronization engine according to the write-in completion notification; and after the synchronization engine sends the target data to the target cluster, a corresponding synchronization completion notification is sent to the metadata service module, so that the metadata service module adds a corresponding mapping relationship in the metadata. According to the method, data synchronization among various object storage clusters can be realized, so that the problem of difficulty in storage capacity expansion of a traditional CDN source station is solved.

The invention relates to the technical field of data management, and discloses a general data management method and system for engineering construction, and the system comprises a meta-model management module, a data dictionary management module, a code table management module, a metadata instantiation module, and a metadata service module. According to the method, based on an aggregation structure of a core metadata model, identification, quality, content, distribution, materials and equipment are forcibly included, the method is responsible for six core sub-models of a unit, and standardized description of engineering construction materials and equipment data is achieved; metadata entities and elements are defined through data dictionary specifications, and semantic ambiguity is reduced; an enumeration value domain is managed in a unified mode in combination with a standardized code table, and it is guaranteed that data are unambiguous; and dynamically adapting to new materials, new attributes and new business scene requirements based on a metadata expansion mechanism. According to the invention, the machine readability and operability of data are improved, and the flexibility and long-term applicability of data management in the full life cycle of a project are improved.

The application discloses a vulnerability determination method and device, electronic equipment and storage medium. The method comprises the following steps: in response to a vulnerability determination instruction, a site request indicating a specified site is determined; when the site request carries a preset abnormal parameter, the site request is updated by using a preset metadata service request information; the updated site request is sent to the specified site to obtain a site response; when the site response carries at least one preset identity credential parameter, a target vulnerability determination result indicating that the metadata service has a vulnerability is obtained. The application can be applied to various scenes such as cloud technology, artificial intelligence, intelligent transportation and auxiliary driving. The application uses the site deployed on the cloud server instance to detect the vulnerability of the metadata service of the cloud server instance, realizes active detection of metadata leakage, and improves the use safety of the cloud service instance. The active detection of metadata leakage disclosed in the application has good adaptability in application.