75 results about "Metadata server" patented technology

The invention discloses a distributed file system metadata index optimization method based on remote direct memory access, which comprises directory metadata index based on path name hash: defining a plurality of layers of directories in a directory tree as key directories, and regarding the path as a character string. After calculation of the hash function, determining the storage position of the data in the memory of the metadata server of the distributed file system; and optimizing the memory position of the file metadata based on the popularity, namely placing the file metadata which is frequently accessed at a position adjacent to the key directory metadata to which the file metadata belongs. A metadata management module in a metadata server of the distributed file system regularly checks the popularity of the file and carries out cold and hot migration according to the situation; and client cache optimization oriented to key directory metadata: a metadata cache of a distributed file system client caches the metadata and directory entries of the key directory. Optimally assuming that the metadata and directory entries of the key directory in the cache are not invalid, and updating the cache according to the result of the next-level metadata search. According to the method, the remote reading performance of the metadata index of the distributed file system is optimized by fully utilizing the characteristic of remote direct memory access, the network transmission frequency is reduced, and the load of the metadata server is effectively reduced.

The invention discloses a single-file concurrent read-write method and system of a distributed file system. The method comprises the steps that S1, a conservative write range lock application request of a preset interval is sent to a metadata server; the range lock takes a block as the minimum granularity; s2, segmenting to-be-written data into a plurality of data blocks according to a preset block length, and applying for a corresponding locking interval in a local memory lock table according to a starting position and an ending position of each data block in a target file; judging whether the currently applied locking interval is overlapped with the registered locking interval or not; s3, if the data block is not in the write range lock, applying for an interval for expanding the write range lock from the metadata server; s4, distributing the segmented data block write-in requests to block engines of corresponding storage nodes, so that the block engines add data block mutual exclusion locks at corresponding positions; and S5, in response to a request of closing or refreshing the target file by the user, sending a request of releasing the write range lock interval. According to the method and the device, the performance is not reduced on the premise of improving the strong consistency.

The invention discloses an IO optimization method and device based on distributed storage, equipment and a storage medium, and relates to the technical field of distributed storage. Comprising the following steps: constructing a distributed metadata index layer between a client and a metadata server, and maintaining a dynamic mapping relationship between a path prefix and the metadata server, so that when the client initiates a file access request, a target metadata server can be directly positioned based on the mapping relationship; and whether the request is routed to the temporary metadata server for load balancing or not is dynamically determined according to the real-time load information of the temporary metadata server. Therefore, the problems of metadata request amplification caused by step-by-step analysis of a tree directory and non-uniform load of a metadata server caused by static directory binding in related technologies are solved; the technical effects that the metadata interaction delay is remarkably reduced, the throughput of the system in massive small file and burst IO scenes is improved, and a single metadata server is prevented from becoming a performance bottleneck are achieved.

The invention relates to a hybrid high-performance storage method and system and a medium. The method comprises the following steps: receiving a data reading or writing request of a client; determining a storage position of the requested data through the metadata server; acquiring or storing the data from a block storage cache layer or an object storage layer according to the storage position; based on a predetermined strategy, dynamically migrating data between the block storage cache layer and the object storage layer, the block storage cache layer being used for storing hot data, and the object storage layer being used for storing cold data; and automatically adjusting the distribution of the data between the block storage cache layer and the object storage layer according to a preset data level factor through an intelligent hierarchical management mechanism. According to the method, low-cost object storage and high-performance block storage are combined, so that the contradiction between the storage performance and the cost in the prior art is solved, the intelligence and the flexibility of data management are improved, and the requirements of different business scenes are met.

The invention discloses a load balancing method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of task scheduling. Comprising the steps of determining a first load balance degree of at least one time step after a current time point of each server node through target data corresponding to each server node in a metadata server cluster and a load prediction model; and determining a sub-tree migration strategy for the metadata server cluster by combining the obtained first load balance degree and the second load balance degree related to the first load index of each server node, and balancing the load of the metadata server cluster through the sub-tree migration strategy. Therefore, the technical problem of how to improve the balancing effect of metadata load balancing in a distributed storage system in related technologies can be solved, and the technical effect of improving the load balancing effect of a metadata server cluster is achieved.

The invention discloses a data processing system, method and equipment, a medium and a program product in the technical field of computers. A gateway, a metadata server and a data server are provided in the same system; the metadata server and the data server can realize separated storage of data and metadata; the transmission delay is shortened by sending the metadata processing request and the data processing request in parallel through the gateway, and the processing delay is shortened by accelerating the processing of the metadata server and the data server, so that the data read-write delay can be reduced.

A method for reducing fraud for SMS OTP authentication includes receiving, by a server from a plurality of participating entities, a plurality of phone numbers and a plurality of metadata associated with a user, and determining, by the server, a phone number tenure for each of the plurality of phone numbers, based on the plurality of metadata. The method further includes determining, by the server, a phone number consistency for the plurality of phone numbers, based on the plurality of metadata, and determining, by the server, a device binding status for each of the plurality of phone numbers, based on the plurality of metadata. The method further includes determining, by the server, a most reliable phone number associated with the user from the plurality of phone numbers, and transmitting, by the server, to the most reliable phone number a SMS OTP, and authenticating the user based on the SMS OTP.

This application provides a management method, apparatus, and metadata server for a distributed storage system. The management method for a distributed storage system provided by this application includes: upon receiving an access request from a client to access a target data file, determining the set of storage nodes where the data blocks corresponding to the target data file are located; searching the storage node set for candidate nodes whose status information is the target status; wherein the target status includes a normal status, a migration-frozen status, and an offline-frozen status; the storage nodes in the distributed storage system are in a migration-frozen status during data migration and in an offline-frozen status after successful data migration; and returning the candidate nodes to the client to instruct the client to obtain the target data file from the candidate nodes. The management method, apparatus, and metadata server for a distributed storage system provided by this application can avoid the problem of existing migration schemes affecting system throughput and causing access delays during data migration.

The invention discloses a distributed lock management method and device, electronic equipment and a storage medium, and relates to the technical field of storage, the management method comprises the following steps: in response to lease overdue of a client, judging whether a server works normally or not; in response to normal work of the server, releasing a distributed lock held by the overdue client, and recording session information of the overdue client into a preset database to prohibit the overdue client from executing lock recovery; and in response to restarting of the server, notifying the metadata server to enter a delay period, determining a target client of which the lock state is not recovered after the delay period is ended, controlling the metadata server to clear lock information of a distributed lock held by the target client, and recording session information of the target client into a preset database, the technical problem that the unconditionally trusted client recovers the lock request after the server is restarted is solved, and the technical effect of guaranteeing the data consistency and reliability of the distributed file system in a fault scene is achieved.

A method and system for identifying IP addresses of an edge device, obtaining associated IP metadata, and storing both the IP address and IP metadata locally is disclosed. The method comprises receiving, by a server, a network message from an edge device that includes an Internet Protocol (IP) address of the edge device, wherein the edge device is not privy to its own IP address. The server extracts the IP address from the network message and obtains associated IP metadata from an IP metadata server. The IP metadata is then transmitted from the server to the edge device for local storage, after which the server deletes the IP metadata to ensure it is not persistently accessible to the server while remaining persistently accessible to the edge device. The edge device utilizes the locally stored IP metadata in executing predefined functionality. The invention also encompasses a system comprising a server, metadata server, and edge device configured to perform these operations. Both the method and system aspects enable efficient IP metadata management while addressing privacy concerns and reducing server-side storage requirements through distributed edge-based storage architecture.

This invention provides a method, apparatus, electronic device, and storage medium for data migration from a storage pool. The storage pool includes a metadata server and an object storage server. The metadata server stores object metadata, including the storage location of the object data. The object storage server stores the object data. The method involves deleting versions of the target object's metadata from the first metadata server, determining the versions to be deleted based on deletion indicators in the target object's metadata from the second metadata server, and then migrating the remaining target object metadata to the second object storage server. Additionally, the method involves updating the storage location in the remaining target object's metadata and writing it in batches to the second metadata server. This achieves the full migration of data from the first storage pool to the second storage pool, avoiding data consistency issues and improving user experience during the full data migration process.

This invention discloses a data management method, apparatus, device, and computer-readable storage medium, relating to the field of distributed storage technology. The method, applied to a distributed network device, includes: receiving traffic data packets of a target data block sent by a distributed storage node; determining the forwarding path of the traffic data packet based on the target flow label in the traffic data packet; if the forwarding path is a secondary verification path corresponding to cross-tenant traffic, accessing a metadata server to obtain the owner tenant identifier from the metadata of the target data block; and discarding the traffic data packet when the requesting tenant identifier in the traffic data packet differs from the owner tenant identifier in the metadata. This invention, by setting the target flow label in the payload header of the traffic data packet, enables the distributed network device to parse the target flow label to detect cross-tenant traffic, thereby triggering secondary verification of cross-tenant traffic, solving the problem of cross-node traffic mixing, and avoiding additional bandwidth overhead.

The invention discloses a data cross-storage device migration method, device and equipment and a storage medium, is applied to a metadata server and relates to the technical field of cloud computing, and the method comprises the following steps: locking metadata of a to-be-migrated data fragment, and sending a migration preparation instruction to a first data server and a second data server, the first data server asynchronously pushes the to-be-migrated data fragments to the second data server based on the migration preparation instruction; when the migration of the to-be-migrated data fragment is completed, determining a target access server from the first data server and the second data server based on a data access request of the client for the to-be-migrated data fragment, so that the client performs data access on the to-be-migrated data fragment based on the target access server; and notifying the first data server to delete the data copy of the to-be-migrated data fragment, and updating the data position of the to-be-migrated data fragment to the second data server. Therefore, seamless migration of the distributed storage system can be supported.

The invention provides a log event replay method and device, electronic equipment and a storage medium, relates to the technical field of distributed file systems, and is used for replaying a log event through a version conflict pre-judgment and multi-stage write-in coordination mechanism. According to the method, the orderliness of write-in operation, the consistency of data and the integrity of transactions in the log event replay process of the distributed file system can be realized. The method comprises the following steps: acquiring a log event from a distributed file system log locally stored in a client; the log event comprises a metadata path, a local version number and local operation information of the target file; querying a synchronous version number of the target file in the distributed file system from a metadata server based on the metadata path; performing version conflict pre-judgment based on the local version number and the synchronous version number; and if no version conflict is pre-judged, executing a multi-stage write operation to replay the log event.

The utility model relates to the technical field of circuit board connecting devices, in particular to an interlayer connector assembly, a stacked connector unit and a data server. The interlayer connector assembly comprises two on-board connectors and a stacked connector group which are respectively connected with the two interconnected circuit boards; the stacked connector group comprises stacked connector units with two opposite plugging ends, the two adjacent stacked connector units are plugged, and the two farthest plugging ends of the stacked connector group are respectively plugged with the two on-board connectors; the stacked connector units have a set unit height to change the stacking height of the interlayer connector assembly to accommodate the spacing between the two interconnected circuit boards when different numbers of stacked connector units are used. By adopting the stacked connector units of one specification and changing the number of the units, various distances between two boards can be supported, and various connectors with different heights do not need to be manufactured, so that the production cost is reduced.

The application provides a method, system, device and storage medium for quickly reading metadata, the method comprising: in response to receiving a metadata request for accessing a first file by a service, a metadata server only reading metadata information of the first file; responding to the metadata request issued by a client according to the read metadata information of the first file; asynchronously issuing the metadata request to a metadata disk and batch loading metadata information of other files in a directory where the first file is located; and in response to again receiving metadata information of a target file in the directory where the first file is located, directly reading metadata information of the target file from the metadata disk. The application improves the metadata cache reading performance of a distributed file storage system in a sequential reading scenario and a random reading scenario. The stored metadata cache space is more efficiently utilized, and the cache hit rate is improved.

The present invention discloses a method for improving MapReduce spill writing based on MPI-IO. The method includes: the Map-side MPI process reads a data set slice from a target file; the Map-side MPI process runs a Map task, performs mapping processing on the data slice and partitions the mapping processing result to obtain partitioned key-value pairs; when it is determined that the size of the mapping processing result exceeds the memory capacity threshold, the Map-side executes a spill writing operation, and parallelly spills the sorted key-value pairs after partitioning to the same disk file to obtain a spill writing processing result; the Reduce-side MPI process pulls the spill writing processing result of the Map-side and uses a Reduce task to perform reduction processing on the key-value pairs to obtain a Reduce processing result; the Reduce-side writes the Reduce processing result to the disk. The present invention aggregates the IO requests of multiple MPI processes by means of parallelly writing a large file, reduces a large amount of file reading and writing, and can also avoid the generation of excessive intermediate files, alleviating the pressure on the metadata server. The present invention can be widely applied to the fields of big data processing frameworks and high-performance computing.

The present application discloses a load balancing method, device, electronic device, storage medium and program product, which relates to the field of task scheduling technology, including: determining the first load balancing degree of each server node in the metadata server cluster at least one time step after the current time point through the target data and load prediction model corresponding to each server node; then combining the obtained first load balancing degree and the second load balancing degree related to the first load index of each server node to jointly determine the subtree migration strategy for the metadata server cluster, and balancing the load of the metadata server cluster through the subtree migration strategy. Therefore, it can solve the technical problem of how to improve the balancing effect of metadata load balancing in a distributed storage system in the related technology, and achieve the technical effect of improving the load balancing effect of the metadata server cluster.

File system management is disclosed. File system metadata is stored in local memory of a client. The portion of local memory storing the file system metadata is under control of the metadata server. The construction and management of the metadata stored in the local memory of the client is driven, initiated and managed by the remote metadata server.

The invention discloses a file processing method and device based on a distributed file system, equipment and a medium. When a client receives an access request of a user for a target file, index node information of the file is obtained from a metadata server of the distributed file system; the client side inquires a mapping table maintained locally in advance according to the first user hash information, and the table records the corresponding relation between the user hash value of the local system and the user identifier. And replacing the first user identifier in the index node information with the queried second user identifier to generate updated index node information. And the updated information is provided for a local operating system, and the operating system performs access permission control based on the mapped user identifier, so that users with the same user name have consistent file access permissions on different clients, and the permission problem caused by inconsistent user identifiers is improved. The file processing method and device can be widely applied to the technical field of file processing based on the distributed file system.

The present disclosure provides a data storage method and related device. The data storage method is applied to a metadata server of a NAS architecture, and includes: based on a received write request, storing small block data to be written in a to-be-written region of a user file in a memory; and in a case where a preset condition is met, storing the memory data after arrangement to a disk of a file system; wherein for the to-be-written region, the memory data after arrangement only includes data existing after a latest operation of the user when the preset condition is met. The present disclosure optimizes the layout of data on the disk, and achieves the purpose of improving the read performance of a client.

The invention discloses a data writing method and a related device, and relates to the technical field of data processing, the method receives an external data writing request, only provides a logic position of to-be-written data, utilizes a metadata server to centrally manage file mapping, prevents each data server from independently maintaining a data copy, and improves the data writing efficiency. And the introduction of the data fragment ID allows the data to be decoupled into manageable fragments, so that the initial data transmission quantity and the size of a transmission unit are reduced. Data to be written are blocked to obtain data blocks to be written, data block names are set for the data blocks to be written, network communication is shared among the data servers, an independent link does not need to be maintained for each data server, the data blocks of the data blocks to be written are only sent to the distributed data servers instead of all the data servers, concurrent repeated data streams are eliminated, and the data processing efficiency is improved. The network congestion risk is reduced, and each data server only stores the required data of the data block to be written, so that the utilization rate of storage resources of the data server is improved.

This system provides advanced vital sign management services for the elderly, along with nutritional and food management, to improve their health, enhance their quality of life, and provide a planned and continuous nutritional management system necessary for extending their healthy lifespan. [Solution] The nutrition management system of this invention comprises a data server 22 that stores health status information D3, including vital signs and other health information of elderly people, acquired at elderly support facilities in shopping districts, and a communication unit 24 that communicates with a terminal 40 connected to the data server 22 via a public communication line 30. The data server 22 stores nutrition management information and / or food management information based on the health status information D3 and information from elderly people obtained via the public communication line. When an elderly person sends a request to the communication unit 24 for information to be provided in order to perform nutrition management, the data server 22 performs a process to provide the information to the terminal 40 from the information stored in the data server 22.

The application discloses an IO optimization method and device based on distributed storage, equipment and a storage medium, and relates to the technical field of distributed storage. The method comprises the following steps: a distributed metadata index layer is constructed between a client and a metadata server, and a dynamic mapping relationship between a path prefix and the metadata server is maintained, so that when the client initiates a file access request, the target metadata server can be directly located based on the mapping relationship, and whether the request is routed to a temporary metadata server for load balancing is dynamically determined according to real-time load information of the target metadata server. Therefore, the problems of metadata request amplification caused by tree directory level-by-level analysis and uneven load of metadata servers caused by static directory binding in the related art are solved, and the technical effects of significantly reducing metadata interaction delay, improving the throughput of the system in the massive small file and burst IO scene, and avoiding a single metadata server from becoming a performance bottleneck are achieved.

The invention discloses a data encryption key updating method and device, equipment and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: periodically traversing data in a storage system according to a preset step length; in the current iteration period, when the encrypted data is traversed according to a preset step length, sending a query instruction to a metadata server to extract version identification information of the first original encryption key; and after the version identification information of the first original encryption key is recorded, continuing to traverse the data in the storage system according to the preset step length until the data in the storage system is traversed, determining that the current iteration period is ended, and entering the next iteration period. The data traversed according to the current preset step length is only encrypted according to the encryption key of the latest version instead of being encrypted once when the key of one version is generated. And compared with the traditional technology, the consumption of computing resources is obviously reduced, and the computing time consumption is greatly shortened.

Generally discussed herein are devices, systems, and methods for cloud resource security. A method can include receiving, at a monitor device and from a first cloud resource of cloud resources hosted by a cloud provider, a request for a token that uniquely identifies the first cloud resource, the request indicating a destination that is a metadata server. The method can include comparing, based on entries in an application programming interface (API) access log, the cloud provider associated with the first cloud resource and a cloud provider associated with the metadata server. The method can include responsive to the cloud provider of the first cloud resource being different from the cloud provider of the metadata server performing a security mitigation action.

The embodiment of the invention discloses a metadata management method, system and device, a server, a storage medium and a program product. The method is applied to a metadata server, the metadata server is located in a distributed system, logs are stored on the metadata server, and the method comprises the steps that a first metadata change event is generated in response to a first metadata change request; and for unoccupied first idle positions in the log, determining a first target position ranked at a preset bit from the first idle positions, and storing the first metadata change event to the first target position, the preset bit being a first bit or a last bit. According to the technical scheme, the maintenance cost in the metadata management process can be reduced.

The application provides a metadata processing method, device, equipment, storage medium and product. The method comprises the following steps: in response to receiving an operation request for metadata in a distributed file system comprising a plurality of preset directory groups sent by a client device, determining a target directory group storing target metadata according to a storage path of the target metadata in the operation request; the target metadata comprises target directory access metadata and / or target file metadata under a directory thereof, and the target directory group is used for storing the target metadata and parent directory timestamp metadata thereof; for each preset directory group, the data in each preset directory group is stored in a corresponding non-uniform memory access node respectively, a plurality of non-uniform memory access nodes are located in corresponding metadata servers, and the data in each preset directory group comprises preset metadata and parent directory timestamp metadata thereof; performing an operation on the target metadata according to an operation type in the operation request, and updating the parent directory timestamp metadata of the target metadata.

This application provides a distributed data reading method, apparatus, and server, relating to the field of data processing. The method includes: a server retrieving multiple file sets from a disk. Each file set may include at least one file. Each file includes metadata for at least one document. The server determines the number of files allocated in each partition based on the quotient of the total number of files in each file set and the number of parallel processes. The server may allocate the files in the file sets to several partitions of parallel processes. Each partition includes at least several allocated files. After determining the number of partitions for the parallel processes, the server may allocate one partition to each parallel process. The parallel process may then read the document corresponding to the metadata in the partition after retrieving it. This method improves data reading efficiency.