134 results about "Metadata server" patented technology

The invention discloses a distributed file system metadata index optimization method based on remote direct memory access, which comprises directory metadata index based on path name hash: defining a plurality of layers of directories in a directory tree as key directories, and regarding the path as a character string. After calculation of the hash function, determining the storage position of the data in the memory of the metadata server of the distributed file system; and optimizing the memory position of the file metadata based on the popularity, namely placing the file metadata which is frequently accessed at a position adjacent to the key directory metadata to which the file metadata belongs. A metadata management module in a metadata server of the distributed file system regularly checks the popularity of the file and carries out cold and hot migration according to the situation; and client cache optimization oriented to key directory metadata: a metadata cache of a distributed file system client caches the metadata and directory entries of the key directory. Optimally assuming that the metadata and directory entries of the key directory in the cache are not invalid, and updating the cache according to the result of the next-level metadata search. According to the method, the remote reading performance of the metadata index of the distributed file system is optimized by fully utilizing the characteristic of remote direct memory access, the network transmission frequency is reduced, and the load of the metadata server is effectively reduced.

The invention discloses a single-file concurrent read-write method and system of a distributed file system. The method comprises the steps that S1, a conservative write range lock application request of a preset interval is sent to a metadata server; the range lock takes a block as the minimum granularity; s2, segmenting to-be-written data into a plurality of data blocks according to a preset block length, and applying for a corresponding locking interval in a local memory lock table according to a starting position and an ending position of each data block in a target file; judging whether the currently applied locking interval is overlapped with the registered locking interval or not; s3, if the data block is not in the write range lock, applying for an interval for expanding the write range lock from the metadata server; s4, distributing the segmented data block write-in requests to block engines of corresponding storage nodes, so that the block engines add data block mutual exclusion locks at corresponding positions; and S5, in response to a request of closing or refreshing the target file by the user, sending a request of releasing the write range lock interval. According to the method and the device, the performance is not reduced on the premise of improving the strong consistency.

The invention relates to the field of distributed storage, in particular to a metadata expansion method and device under a distributed file system, equipment and a storage medium. The method comprises the following steps: receiving a capacity expansion cluster request sent by a client; according to the capacity expansion cluster request, a to-be-migrated target process is determined, the target process is used for managing metadata, and an identifier of a file or directory where the metadata is located and the target process have a Hash mapping relation; metadata of the target process is copied from the first metadata server to a second metadata server, the Hash mapping relation is adjusted, and the second metadata server is different from the first metadata server. According to the embodiment of the invention, dynamic scheduling and load balancing are carried out on the metadata server cluster through hash based on the directory or the file, and the influence on services is remarkably reduced by minimizing the data migration cost. Compared with a traditional method, the load balancing capacity is better, the data migration cost is lower, and the expansibility is higher.

The invention discloses an IO optimization method and device based on distributed storage, equipment and a storage medium, and relates to the technical field of distributed storage. Comprising the following steps: constructing a distributed metadata index layer between a client and a metadata server, and maintaining a dynamic mapping relationship between a path prefix and the metadata server, so that when the client initiates a file access request, a target metadata server can be directly positioned based on the mapping relationship; and whether the request is routed to the temporary metadata server for load balancing or not is dynamically determined according to the real-time load information of the temporary metadata server. Therefore, the problems of metadata request amplification caused by step-by-step analysis of a tree directory and non-uniform load of a metadata server caused by static directory binding in related technologies are solved; the technical effects that the metadata interaction delay is remarkably reduced, the throughput of the system in massive small file and burst IO scenes is improved, and a single metadata server is prevented from becoming a performance bottleneck are achieved.

The invention belongs to the technical field of Lustre file management, and particularly relates to a method and system for improving Lustre small file management and performance, and the system comprises a nested file system, a three-level index module, a cache index module and a self-adaptive cache module; the nested file system realizes management and storage of small files; the three-level index module comprises a small file global metadata index, a container file internal index and a cache index and is used for optimizing retrieval and storage management of small files; the cache index module is used for managing small files accessed at high frequency and reducing repeated I / O requests; and according to the self-adaptive caching module, the client dynamically adjusts a caching strategy of a small file according to the file access frequency and mode, so that the reading speed of a high-frequency access file is improved, and network I / O (Input / Output) operation is reduced. The performance bottleneck of a Lustre file system during small file processing is solved through a nested file system, and the access burden of a metadata server and an object storage server is reduced through a three-level index architecture.

The invention relates to a file processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: receiving a file operation request, wherein the file operation request comprises an operation path; sending the operation path to a metadata server, and receiving node information of a target node corresponding to the operation path returned by the metadata server; the target node is a node in a file cluster tree; and processing a file under the target node corresponding to the file operation request according to the node information of the target node. By adopting the method, the file operation efficiency can be improved.

The invention relates to a hybrid high-performance storage method and system and a medium. The method comprises the following steps: receiving a data reading or writing request of a client; determining a storage position of the requested data through the metadata server; acquiring or storing the data from a block storage cache layer or an object storage layer according to the storage position; based on a predetermined strategy, dynamically migrating data between the block storage cache layer and the object storage layer, the block storage cache layer being used for storing hot data, and the object storage layer being used for storing cold data; and automatically adjusting the distribution of the data between the block storage cache layer and the object storage layer according to a preset data level factor through an intelligent hierarchical management mechanism. According to the method, low-cost object storage and high-performance block storage are combined, so that the contradiction between the storage performance and the cost in the prior art is solved, the intelligence and the flexibility of data management are improved, and the requirements of different business scenes are met.

The invention discloses a load balancing method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of task scheduling. Comprising the steps of determining a first load balance degree of at least one time step after a current time point of each server node through target data corresponding to each server node in a metadata server cluster and a load prediction model; and determining a sub-tree migration strategy for the metadata server cluster by combining the obtained first load balance degree and the second load balance degree related to the first load index of each server node, and balancing the load of the metadata server cluster through the sub-tree migration strategy. Therefore, the technical problem of how to improve the balancing effect of metadata load balancing in a distributed storage system in related technologies can be solved, and the technical effect of improving the load balancing effect of a metadata server cluster is achieved.

The invention discloses a data processing system, method and equipment, a medium and a program product in the technical field of computers. A gateway, a metadata server and a data server are provided in the same system; the metadata server and the data server can realize separated storage of data and metadata; the transmission delay is shortened by sending the metadata processing request and the data processing request in parallel through the gateway, and the processing delay is shortened by accelerating the processing of the metadata server and the data server, so that the data read-write delay can be reduced.

The invention belongs to the field of big data, and relates to a directory path analysis method, which comprises the following steps: acquiring directory information of a file system, and creating a metadata cache in a metadata client for storing the directory information; receiving a directory path analysis request, and judging whether a directory accessed by the directory path analysis request exists in the metadata cache or not based on the directory information; if the directory accessed by the directory path analysis request exists in the metadata cache, acquiring an entity identifier corresponding to the directory from the metadata cache; sending the entity identifier to a metadata server, so as to query result data corresponding to the directory from a key value database through the metadata server; and returning result data corresponding to the directory path analysis request. The invention further provides a device, equipment and a medium. In addition, the invention also relates to a block chain technology, and directory information and result data can be stored in a block chain. According to the method, the directory path analysis time delay can be reduced, and the problem of near root hotspots is solved.

The present disclosure is associated with operation of a metadata server for package management including development and deployment of software. To illustrate, a metadata server stores a database model including multiple data structures that each include one or more entries of multiple data objects and a set of mapping rules defining links between different data structures of the multiple data structure. A method of operating the metadata server includes determining access permissions for an entity. The access permissions correspond to authorization to access a set of the multiple data structures. The method further includes identifying and executing a mapping rule to authorize a link between two or more data structures. The method further includes accessing a portion of requested metadata from the two or more data structures that have an authorized active link, and providing an output of the portion of the requested metadata arranged according to an output format.

A method for reducing fraud for SMS OTP authentication includes receiving, by a server from a plurality of participating entities, a plurality of phone numbers and a plurality of metadata associated with a user, and determining, by the server, a phone number tenure for each of the plurality of phone numbers, based on the plurality of metadata. The method further includes determining, by the server, a phone number consistency for the plurality of phone numbers, based on the plurality of metadata, and determining, by the server, a device binding status for each of the plurality of phone numbers, based on the plurality of metadata. The method further includes determining, by the server, a most reliable phone number associated with the user from the plurality of phone numbers, and transmitting, by the server, to the most reliable phone number a SMS OTP, and authenticating the user based on the SMS OTP.

The invention relates to a file lock management method and device, computer equipment, a medium and a product. The method is applied to a file lock management node, the file lock management node, a metadata server and a first client are in pairwise communication connection, and the method comprises the following steps: when it is detected that communication between the metadata server and the first client is abnormal, obtaining a file lock according to a file lock obtaining request sent for the first client; generating file lock management information of a target file lock corresponding to the first client, wherein the file lock acquisition request is sent to a file lock management node by a second client; sending file lock management information to the first client, so that the first client executes management operation based on the file lock management information to obtain a file lock operation result; and after a file lock operation result sent by the first client is received, generating an acquisition request result of the target file lock according to the file lock operation result, and sending the acquisition request result to the second client. By adopting the method, the management self-healing performance of file lock management is improved.

This application provides a management method, apparatus, and metadata server for a distributed storage system. The management method for a distributed storage system provided by this application includes: upon receiving an access request from a client to access a target data file, determining the set of storage nodes where the data blocks corresponding to the target data file are located; searching the storage node set for candidate nodes whose status information is the target status; wherein the target status includes a normal status, a migration-frozen status, and an offline-frozen status; the storage nodes in the distributed storage system are in a migration-frozen status during data migration and in an offline-frozen status after successful data migration; and returning the candidate nodes to the client to instruct the client to obtain the target data file from the candidate nodes. The management method, apparatus, and metadata server for a distributed storage system provided by this application can avoid the problem of existing migration schemes affecting system throughput and causing access delays during data migration.

The invention discloses a distributed lock management method and device, electronic equipment and a storage medium, and relates to the technical field of storage, the management method comprises the following steps: in response to lease overdue of a client, judging whether a server works normally or not; in response to normal work of the server, releasing a distributed lock held by the overdue client, and recording session information of the overdue client into a preset database to prohibit the overdue client from executing lock recovery; and in response to restarting of the server, notifying the metadata server to enter a delay period, determining a target client of which the lock state is not recovered after the delay period is ended, controlling the metadata server to clear lock information of a distributed lock held by the target client, and recording session information of the target client into a preset database, the technical problem that the unconditionally trusted client recovers the lock request after the server is restarted is solved, and the technical effect of guaranteeing the data consistency and reliability of the distributed file system in a fault scene is achieved.

A method and system for identifying IP addresses of an edge device, obtaining associated IP metadata, and storing both the IP address and IP metadata locally is disclosed. The method comprises receiving, by a server, a network message from an edge device that includes an Internet Protocol (IP) address of the edge device, wherein the edge device is not privy to its own IP address. The server extracts the IP address from the network message and obtains associated IP metadata from an IP metadata server. The IP metadata is then transmitted from the server to the edge device for local storage, after which the server deletes the IP metadata to ensure it is not persistently accessible to the server while remaining persistently accessible to the edge device. The edge device utilizes the locally stored IP metadata in executing predefined functionality. The invention also encompasses a system comprising a server, metadata server, and edge device configured to perform these operations. Both the method and system aspects enable efficient IP metadata management while addressing privacy concerns and reducing server-side storage requirements through distributed edge-based storage architecture.

The invention provides a file lock processing method and device, electronic equipment and a storage medium. Under the condition that a file stream client obtains a first write request for a target file submitted by a user, and under the condition that the target file is not locked, a metadata server stores a corresponding relation among the ID of the file stream client, a first locking serial number and the ID of the target file in a persistent storage space. If the metadata server side breaks down (or crashes), the metadata server side does not lock the target file, and the file stream client side does not receive a locking response which is sent by the metadata server side and is used for indicating that the target file is locked, so that the file stream client side does not store information which is used for indicating that the target file is locked. In this way, the file stream client and other normal metadata servers can be consistent in cognition of whether the target file is locked or not, and deadlock is avoided.

This invention provides a method, apparatus, electronic device, and storage medium for data migration from a storage pool. The storage pool includes a metadata server and an object storage server. The metadata server stores object metadata, including the storage location of the object data. The object storage server stores the object data. The method involves deleting versions of the target object's metadata from the first metadata server, determining the versions to be deleted based on deletion indicators in the target object's metadata from the second metadata server, and then migrating the remaining target object metadata to the second object storage server. Additionally, the method involves updating the storage location in the remaining target object's metadata and writing it in batches to the second metadata server. This achieves the full migration of data from the first storage pool to the second storage pool, avoiding data consistency issues and improving user experience during the full data migration process.

This invention discloses a data management method, apparatus, device, and computer-readable storage medium, relating to the field of distributed storage technology. The method, applied to a distributed network device, includes: receiving traffic data packets of a target data block sent by a distributed storage node; determining the forwarding path of the traffic data packet based on the target flow label in the traffic data packet; if the forwarding path is a secondary verification path corresponding to cross-tenant traffic, accessing a metadata server to obtain the owner tenant identifier from the metadata of the target data block; and discarding the traffic data packet when the requesting tenant identifier in the traffic data packet differs from the owner tenant identifier in the metadata. This invention, by setting the target flow label in the payload header of the traffic data packet, enables the distributed network device to parse the target flow label to detect cross-tenant traffic, thereby triggering secondary verification of cross-tenant traffic, solving the problem of cross-node traffic mixing, and avoiding additional bandwidth overhead.

The invention discloses a data cross-storage device migration method, device and equipment and a storage medium, is applied to a metadata server and relates to the technical field of cloud computing, and the method comprises the following steps: locking metadata of a to-be-migrated data fragment, and sending a migration preparation instruction to a first data server and a second data server, the first data server asynchronously pushes the to-be-migrated data fragments to the second data server based on the migration preparation instruction; when the migration of the to-be-migrated data fragment is completed, determining a target access server from the first data server and the second data server based on a data access request of the client for the to-be-migrated data fragment, so that the client performs data access on the to-be-migrated data fragment based on the target access server; and notifying the first data server to delete the data copy of the to-be-migrated data fragment, and updating the data position of the to-be-migrated data fragment to the second data server. Therefore, seamless migration of the distributed storage system can be supported.

The invention provides a log event replay method and device, electronic equipment and a storage medium, relates to the technical field of distributed file systems, and is used for replaying a log event through a version conflict pre-judgment and multi-stage write-in coordination mechanism. According to the method, the orderliness of write-in operation, the consistency of data and the integrity of transactions in the log event replay process of the distributed file system can be realized. The method comprises the following steps: acquiring a log event from a distributed file system log locally stored in a client; the log event comprises a metadata path, a local version number and local operation information of the target file; querying a synchronous version number of the target file in the distributed file system from a metadata server based on the metadata path; performing version conflict pre-judgment based on the local version number and the synchronous version number; and if no version conflict is pre-judged, executing a multi-stage write operation to replay the log event.

The invention discloses a metadata balance management method and device, equipment and a medium, and relates to the technical field of data management, and the method comprises the steps: carrying out the fragmentation of a local preset file sharing directory based on a received configuration instruction, so as to obtain a plurality of corresponding fragmentation directories and a plurality of corresponding fragmentation values; performing Hash calculation on the plurality of fragment values and directory numbers corresponding to a preset file sharing directory, so as to determine metadata server daemon process numbers corresponding to the plurality of fragment directories through the obtained plurality of Hash values; respectively distributing a target daemon process and a standby daemon process for the plurality of fragment directories; and if the target daemon process is abnormal, starting a standby daemon process corresponding to the target daemon process. Therefore, the load balancing of the metadata server can be efficiently realized, so that the performance and the stability of the distributed file system are improved.

The present application discloses a service quality control method, apparatus, device and storage medium, which relates to the technical field of distributed file system storage. Applied to the server, the method includes: setting service quality information of a service directory of a current node in the distributed file system; receiving, by a metadata server in the server, a detection result of performing hierarchical service quality detection on the service directory based on the service quality information when a client performs a service read / write operation, so as to determine a target service directory that needs to perform service quality control in the service directory according to the detection result; detecting target service quality information corresponding to the target service directory, so as to delete the target service directory according to the target service quality information. Through the technical solution of the present application, the service quality of the service directory can be effectively controlled, thereby ensuring the storage quality of the overall distributed file system storage service.

The utility model relates to the technical field of circuit board connecting devices, in particular to an interlayer connector assembly, a stacked connector unit and a data server. The interlayer connector assembly comprises two on-board connectors and a stacked connector group which are respectively connected with the two interconnected circuit boards; the stacked connector group comprises stacked connector units with two opposite plugging ends, the two adjacent stacked connector units are plugged, and the two farthest plugging ends of the stacked connector group are respectively plugged with the two on-board connectors; the stacked connector units have a set unit height to change the stacking height of the interlayer connector assembly to accommodate the spacing between the two interconnected circuit boards when different numbers of stacked connector units are used. By adopting the stacked connector units of one specification and changing the number of the units, various distances between two boards can be supported, and various connectors with different heights do not need to be manufactured, so that the production cost is reduced.

The application provides a method, system, device and storage medium for quickly reading metadata, the method comprising: in response to receiving a metadata request for accessing a first file by a service, a metadata server only reading metadata information of the first file; responding to the metadata request issued by a client according to the read metadata information of the first file; asynchronously issuing the metadata request to a metadata disk and batch loading metadata information of other files in a directory where the first file is located; and in response to again receiving metadata information of a target file in the directory where the first file is located, directly reading metadata information of the target file from the metadata disk. The application improves the metadata cache reading performance of a distributed file storage system in a sequential reading scenario and a random reading scenario. The stored metadata cache space is more efficiently utilized, and the cache hit rate is improved.

A data management method and apparatus for a distributed storage system, and a device and a medium, which relate to the technical field of computers. The method is applied to a task tracking management module in a distributed storage system, which comprises a cascaded switch, a plurality of leaf switches, a plurality of clients, a plurality of servers, a metadata server and the task tracking management module, wherein all the leaf switches are connected to the cascaded switch, each leaf switch is connected to the plurality of clients and the plurality of servers, the metadata server is connected to one of the leaf switches, and the task tracking management module is connected to one of the leaf switches. The method comprises: upon receiving a target task, determining a target server where data corresponding to the target task is located (S101); and determining candidate clients which are connected to the same leaf switch as the target server, and selecting, from among the candidate clients, a target client for executing the target task, so as to use the target client to execute the target task (S102). Therefore, the performance of a blocking network is improved.

The present invention discloses a method for improving MapReduce spill writing based on MPI-IO. The method includes: the Map-side MPI process reads a data set slice from a target file; the Map-side MPI process runs a Map task, performs mapping processing on the data slice and partitions the mapping processing result to obtain partitioned key-value pairs; when it is determined that the size of the mapping processing result exceeds the memory capacity threshold, the Map-side executes a spill writing operation, and parallelly spills the sorted key-value pairs after partitioning to the same disk file to obtain a spill writing processing result; the Reduce-side MPI process pulls the spill writing processing result of the Map-side and uses a Reduce task to perform reduction processing on the key-value pairs to obtain a Reduce processing result; the Reduce-side writes the Reduce processing result to the disk. The present invention aggregates the IO requests of multiple MPI processes by means of parallelly writing a large file, reduces a large amount of file reading and writing, and can also avoid the generation of excessive intermediate files, alleviating the pressure on the metadata server. The present invention can be widely applied to the fields of big data processing frameworks and high-performance computing.

A method of generating a lakehouse metadata service log, a method of querying a lakehouse metadata service log, an electrode device and a storage medium are provided. The method of generating a lakehouse metadata service log includes: after a metadata server receives a data processing request sent by a data engine, determining, by the metadata server, first request identification information based on the data processing request, such that the first request identification information is used for identifying the data processing request; and then executing, by the metadata server, the request processing logic corresponding to the data processing request, and generating a log corresponding to the request processing logic based on the first request identification information, such that the log carries the first request identification information.

The present application discloses a load balancing method, device, electronic device, storage medium and program product, which relates to the field of task scheduling technology, including: determining the first load balancing degree of each server node in the metadata server cluster at least one time step after the current time point through the target data and load prediction model corresponding to each server node; then combining the obtained first load balancing degree and the second load balancing degree related to the first load index of each server node to jointly determine the subtree migration strategy for the metadata server cluster, and balancing the load of the metadata server cluster through the subtree migration strategy. Therefore, it can solve the technical problem of how to improve the balancing effect of metadata load balancing in a distributed storage system in the related technology, and achieve the technical effect of improving the load balancing effect of the metadata server cluster.