39 results about "Distributed File System" patented technology

This invention relates to the field of storage system operation and maintenance and optimization technology, and in particular to a method, system, and apparatus for write operation traffic statistics based on kernel mounting. The method includes establishing a mapping relationship between all directories in a distributed file system and clients; identifying at least one target directory accessed by clients within a preset time period; monitoring each target directory, obtaining the storage capacity change value of the target directory within the preset time period, and the number of file handle accesses to the target directory; calculating the write operation traffic of each client to the target directory based on the storage capacity change value and the number of file handle accesses; and implementing at least one optimization strategy based on the write operation traffic of each client to the target directory to improve the stability of the distributed file system. This invention can improve the stability of a distributed file system while using kernel mounting to access it.

This invention discloses a software data analysis method and system based on cloud computing, belonging to the field of cloud computing technology. The method includes the following steps: collecting access records of data shards in a distributed file system, constructing a business-related access log, and calculating the access heat index and periodic feature vector of the data shards; extracting the target data shard identifier and matching the target data shard identifier with the periodic feature vector; scheduling a background thread to stream the prefetched target shard from the cold storage medium to the hot storage buffer, and adjusting the corresponding I / O scheduling priority according to the preheating state. This invention solves the problem in existing technologies where static cold / hot layered architectures cannot perceive sudden changes in business access patterns, leading to delayed cold data determination, cross-layer queries triggering data migration storms, and consequently, performance fluctuations in hot data queries. This is achieved by constructing a business-aware prediction-prefetch-feedback closed-loop mechanism.

The application discloses an Internet of Things mass service bill processing system, method and device and a storage medium, and comprises an application software service layer, a platform environment service layer and an infrastructure service layer.The application software service layer is used for collecting first service bills, and performing format analysis, deduplication, merging and distribution processing on the first service bills to obtain first bill files and first bill messages, and then uploading the first bill files and the first bill messages.The platform environment service layer is deployed with a distributed file system, a distributed coordination service, a data database and a streaming processing framework, is used for receiving the first bill files and the first bill messages, and performing full-amount synchronization on the data database according to the first bill files and incremental synchronization on the data database according to the first bill messages.The infrastructure service layer comprises a bottom cloud host, and is used for providing a computing resource pool, a storage resource pool and a network resource pool.The application improves the processing efficiency of service bills, optimizes the use perception of users, and can be widely applied to the technical field of data processing.

In some examples, a data management system processes snapshots of a distributed file system, the distributed file system having files, each file comprising multiple data chunks. The data management system performs operations including storing file-to-chunk mapping in file system metadata; creating, for each chunk, a chunk generation ID by associating each chunk with a file system generation ID; in a next-generation snapshot of the distributed file system, incrementing, for all chunks in the next-generation snapshot, the respective chunk generation IDs; and taking a snapshot of the file system metadata and storing an updated file-to-chunk mapping associated with the next-generation snapshot.

This invention discloses a blockchain-based trusted traceability system and method for the entire lifecycle of medical samples, belonging to the fields of medical sample management, blockchain, and edge computing technologies. The system includes a data acquisition unit, an edge verification unit, a blockchain evidence storage unit, and an intelligent control and interaction unit. The data acquisition unit implements unique sample identification binding and full-process data acquisition, supporting local caching even when offline. The edge verification unit performs data validity verification and anomaly filtering. The blockchain evidence storage unit adopts a consortium blockchain architecture, storing hash digests, dual timestamps, and digital signatures on-chain, while original sensitive data is encrypted and stored in an off-chain distributed file system. The intelligent control and interaction unit implements access control, traceability query, and anomaly output. This invention achieves trusted traceability and privacy compliance for the entire lifecycle of medical samples through real-time edge filtering, off-chain encryption and on-chain hashing, and offline dual timestamp calibration.

The application discloses a kind of write flow control method and device of distributed file system, electronic equipment, it is related to distributed storage technical field or other related fields, including: according to preset sliding window, the processing time length of each storage node recent N times history file write request is recorded, and time delay sequence is obtained;Based on the average value of time delay sequence and preset reference time delay, node type is divided into high time delay node or low time delay node, and respectively determine the flow control coefficient of node;For each file write request to be processed, the global control coefficient is calculated based on the flow control coefficient corresponding to all storage nodes involved in request;Any random number in preset range is obtained, in the case where random number is less than global control coefficient, file write request is sent to storage node, otherwise refuse to send.The application solves the technical problems that distributed file system in related art cannot identify and inhibit single slow storage node, is easy to cause retry avalanche, and then drags down global service.

In a fault-tolerant method for event information delivery within a file system cluster, one or more processors determine event information associated with file system activity performed by nodes of the cluster. The one or more processors add the event information to an event log buffer in memory. The one or more processors receive a first log sequence number (LSN) associated with flushing recovery information from a recovery log buffer. The one or more processors determine event information in the event log buffer that has a log sequence number less than or equal to the first log sequence number, and determine that the event information includes a log sequence number less than or equal to the first log sequence number, and the one or more processors flush the corresponding event information from the event log buffer to disk storage.

Methods, systems, and devices for staging of non-fungible tokens (NFTs) before deployment of the NFTs are described. A custodial token platform receives user inputs corresponding to a collection of NFTs associated with a smart contract to be stored on a blockchain distributed data store. The custodial token platform deploys the smart contract to the blockchain distributed data store based on the user inputs, receives data corresponding to the collection, and uploads the data corresponding to the collection to a distributed file system. Uploading the data corresponding to the collection includes uploading first content for a first NFT and uploading second content for a second NFT to the distributed file system. The custodial token platform receives an update to the first content, uploads the updated first content to the distributed file system, and updates the smart contract to include indications of the data corresponding to the collection and the updated first content.

This application discloses a distributed lease management method and system, storage medium, and electronic device, relating to the field of storage management technology. It is used at the client layer and includes: upon receiving a file operation request, sending a lease request to a node layer, wherein the lease request indicates a target lease for the target file corresponding to the file operation request; after receiving the target lease sent by the node layer, accessing the target file in the storage system according to the target lease, wherein the node layer corresponds to at least one client layer; in the event of a conflict between the target lease and a reference node layer, the node layer accesses the cluster layer; and upon receiving an authorization instruction sent by the cluster layer, the node layer sends the target lease. This solves the technical problem of low resource utilization in distributed file systems in related technologies, achieving the technical effect of improving file system resource utilization.

This disclosure provides a database index generation method and apparatus, a data processing method and apparatus, relating to the field of computer technology, specifically to distributed systems, storage management, and other technical fields. The specific implementation scheme is as follows: based on the metadata of the metadata database in the distributed file system, determine the items to be aggregated related to file quotas; based on the items to be aggregated, determine the item aggregation function; create an aggregation index in the metadata database; perform statistics on the items to be aggregated according to the item aggregation function to obtain statistical data; and store the statistical data as aggregated data in the aggregation index.

The present application belongs to the technical field of storage. Provided are a data processing method and apparatus in a distributed file system, and a storage cluster. In the method, on the basis that a distributed file system stores a log of a file in a cache space, persistent storage is performed on the log on the basis of whether the data volume of the log in the cache space meets a condition. During this process, when the data volume of a certain log does not meet the condition, the log is redirected to release an initial position of the log in the cache space, such that the distributed file system can continue to process the next log of the log. Therefore, the possibility of aggregating discrete data over a relatively long time span is provided. The method is applicable to any type of file. Moreover, since the persistent storage process is performed on the basis of the data volume of the log, the log which meets the condition is persistently stored, thereby avoiding the fragmentation of a persistent space, and improving the access performance of the system.

This invention discloses a file operation method and file operating system for distributed file systems. One embodiment of the file operation method includes: in response to the startup of a target business process, loading a dynamically linked library configured based on the LD_PRELOAD mechanism, and obtaining a globally unique process access identifier allocated by the user-space distributed file system server cluster; intercepting system calls related to glibc library files, allocating a unique file descriptor within the process in user space to replace the Linux kernel allocation logic, concatenating the process access identifier and the file descriptor to generate a globally unique file handle, and calling the server cluster interface to complete the file operation; and notifying the server cluster to release access resources when the process exits. This invention relates to the field of distributed storage technology, solves the performance bottleneck problem of existing user-space file systems, effectively eliminates kernel redundancy overhead, and enables an order-of-magnitude improvement in the read and write performance of distributed file systems.

Embodiments of the present application provide a snapshot processing method, device, storage medium and program product. In the snapshot data processing method, after uploading the snapshot data corresponding to the current snapshot operation of the target storage volume, the write-in node can delete the shadow file corresponding to the target storage volume in the case that there is no unmerged update data in the target storage volume. Before deleting the shadow file, the first write-in node can create a snapshot mutual exclusion identifier in the target storage volume, which can block the next snapshot operation of the target storage volume within a set time range. Further, other write-in nodes in the distributed file system cannot perform snapshot operation on the target storage volume within the set time range, reducing the probability that the shadow file to be deleted by the write-in node is reused by the next snapshot operation, thereby reducing the risk of data loss caused by the deletion operation of the write-in node on the shadow file, and being beneficial to improving the accuracy of the update data written into the target storage volume during the next snapshot.

This invention discloses a semantic tracing and lazy reclamation method for orphan files in a distributed file system, belonging to the field of distributed file system technology. This method persists directory entry change operations through full-link logging and establishes an independent operation log index for each inode. It monitors changes in the inode hard link count in real time, marks files to be classified as orphan files when the hard link count reaches zero, and triggers the semantic tracing process. It traces back the file's entire lifecycle relationships to complete the classification, matches the corresponding hierarchical reclamation strategy based on the classification results, and generates a read-only snapshot before releasing space when the reclamation trigger conditions are met. This method can accurately distinguish different types of orphan files, reduce resource consumption during the reclamation process, avoid accidental deletion of business data, balance storage space reclamation efficiency and data security, and improve the operational stability of the distributed file system.

Systems and methods are disclosed to collect file data for log files associated with a distributed file system, the file data including size values and time values associated with the log files, segment the log files based on the time values to create time-based partitions, identify historical patterns in the file data, and optimize a data retention policy for the distributed file system based on the historical patterns.

The embodiment of the present disclosure discloses a data transmission method and system between a remote client and a Hive table and a storage medium. The method comprises: in response to a data transmission instruction triggered by a user, the remote client modifies the data transmission instruction into a data relay transmission instruction and sends it to a Hive server, wherein the data relay transmission instruction comprises a relay storage path under a distributed file system, and the relay storage path is used to temporarily store the to-be-transmitted data in the process of data transmission between the remote client and the Hive table; the Hive server analyzes and compiles the data relay transmission instruction, generates a data transmission execution plan and sends it to a Yarn cluster; based on the relay storage path and the data transmission execution plan, the remote client and the Yarn cluster cooperatively execute the data transmission operation to complete the transmission of the to-be-transmitted data between the remote client and the Hive table. The technical scheme of the present disclosure enables the user to trigger the instruction only once, thereby realizing the data transmission between the remote client and the Hive table.

The application discloses a distributed file system access performance improvement method and system, comprising the following steps: firstly, receiving a client access request to form a to-be-processed request sequence; performing access mode analysis on the sequence to obtain an access mode category and a related first access characteristic item, and meanwhile, resolving a second access characteristic item containing suppression characteristic information and a corresponding suppression scope; determining a target access characteristic item belonging to the suppression scope from the first access characteristic item, adjusting the access mode category in combination with the second characteristic item, and obtaining a target access mode judgment result; determining a file system performance optimization type according to the result, generating and triggering an execution corresponding resource configuration or strategy adjustment instruction. The method improves the optimization strategy adaptability by combining the access mode and the suppression characteristic optimization mode judgment, and effectively enhances the distributed file system access performance.

An apparatus includes a processing device configured to receive a request to store one or more data portions at a given node of a distributed file system cluster and monitor performance parameters of each storage device associated with the given node, the performance parameters including a first performance parameter characterizing a frequency of data access and at least a second performance parameter characterizing available space. The processing device is further configured to determine an overall performance indicator for each of the storage devices associated with the given node based at least in part on the monitored performance parameters, and select at least one of the storage devices associated with the given node to store the one or more data portions based at least in part on the overall performance indicators.

The invention provides a data processing method and device in a distributed file system and a storage cluster, and belongs to the technical field of storage. According to the method, on the basis that a distributed file system stores logs of files into a cache space, persistent storage is carried out on the logs on the basis that whether the data size of the logs in the cache space meets conditions or not, and in the process, when the data size of a certain log does not meet the conditions, the log is stored in the cache space in a persistent mode. The log is redirected to release the initial position of the log in the cache space, so that the distributed file system can continue to process the next log of the log, the probability of aggregation of discrete data in a long time span is provided, the method is suitable for any type of files, and the method is easy to implement. The persistent storage process is carried out based on the data volume of the logs, so that the logs meeting the conditions can be stored persistently, fragmentation of a persistent space is avoided, and the access performance of the system is improved.

The application relates to a distributed file system based on content semantic hash and belongs to the technical field of data storage. The system breaks through the retrieval and storage limitation of traditional file systems depending on file names or file paths by means of a content semantic hash technology, realizes semantic-driven management of data in combination with a distributed cluster architecture, and solves problems such as high redundancy of heterogeneous data, low retrieval precision, and low efficiency of storage resource scheduling. The system takes a hash index layer as a core support, extracts deep semantic features of files through a semantic processing layer, maps the features to generate unique semantic hash values by the hash index layer, and realizes intelligent deduplication, semantic clustering storage and accurate retrieval on the basis.

The application provides a blockchain data sharing method and system based on identity signature and SM4, and belongs to the technical field of data encryption. The method comprises the following steps: at a data collection end, performing certificateless identity authentication on an original file to be shared; performing encryption on the original file by using a national secret SM4 symmetric encryption algorithm to obtain an encrypted file; uploading the encrypted file to an IPFS distributed file system to obtain a corresponding content identifier CID; taking a device identity identifier, an original file fingerprint, a digital signature and the CID in the minimum signature package as anchor point information, and writing the anchor point information into a blockchain based on a DPOS consensus mechanism; and reading the anchor point information from the blockchain by a verification party, downloading the encrypted file from the IPFS according to the CID, and decrypting to obtain the original file. The application realizes trusted on-chain data, safe transmission and controllable sharing.

The application belongs to the technical field of unmanned aerial vehicle application data processing, and particularly relates to a multi-modal data storage and information retrieval system and method, which comprises: a data access and preprocessing layer, receiving multi-modal data from different platforms and preprocessing, mapping the preprocessed multi-modal data into a high-dimensional semantic space by using a multi-modal large model to generate a unified semantic feature vector; a data storage and index layer, storing the original multi-modal data and the unified semantic feature vector thereof into a relational database, a distributed file system, a vector database or an in-memory database respectively, and constructing a multi-level index structure according to the stored multi-modal data; an application service layer, used for user interaction, receiving a retrieval instruction, and displaying a retrieval result; a retrieval and correlation analysis layer, performing retrieval based on the retrieval instruction, and supporting correlation analysis on three index results of a vector index, a structured index and a space-time index, generating a correlation result and fusing and sorting the index results.

The application provides a vector retrieval method and device, electronic equipment and a storage medium, wherein the method comprises: obtaining a vector retrieval request from a client through a coordination node, analyzing the vector retrieval request, and generating at least one corresponding execution plan; distributing each execution plan to a work node corresponding to each execution plan in a distributed file system; each work node responds to the corresponding execution plan, obtains a candidate vector index meeting the requirement of the vector retrieval request from the distributed file system; a vector module uses a target index library corresponding to the candidate vector index, determines N specified vectors with the highest similarity to a target vector indicated by the vector retrieval request through searching in all candidate vector indexes, wherein N is an integer greater than or equal to 1; and the coordination node returns the N specified vectors to the client. The application solves the problems of low efficiency and poor scalability of vector retrieval in the related art.

This application provides a call detail record (CDR) data processing method, apparatus, electronic device, and storage medium. The method includes: acquiring CDR data, which includes uplink data and downlink data, wherein the downlink data includes a submission form and a corresponding receipt; determining a message identifier for the CDR data based on the submission form and the corresponding receipt; setting the message identifier as an identifier for a key-value pair and setting the CDR data as the specific value of the key-value pair; and then caching the key-value pair in an in-memory database, which is used to retrieve the CDR data corresponding to the message identifier. In the above implementation, since the in-memory database is directly stored in memory, and the access speed of the in-memory database is much faster than that of a distributed file system, the data user can retrieve the CDR data corresponding to the message identifier almost in real time, thereby effectively improving the real-time performance of associating CDR data.

The embodiment of the application provides a method, device and equipment supporting multiple instances, the method comprises the following steps: receiving an instance creation request, wherein the instance creation request comprises the identification of the instance; determining the identification of the directory tree corresponding to the instance according to the instance creation request; creating the directory tree in the device group, and generating the mapping relationship between the directory tree and the device group. Since multiple directory trees can be created in any device group, compared with only one directory tree in one device group, the flexibility of file storage in the distributed file system is improved.

This invention discloses a distributed metadata ordered synchronization method based on inode semantics, belonging to the field of distributed file system technology. This method parses the semantic dependencies of inode metadata operations, constructs a directed acyclic graph of metadata operation dependencies, and groups operations with semantic dependencies into a serial synchronization queue, while operations without dependencies and belonging to different directory tree branches into a parallel synchronization queue. A queue-level lock is allocated to the serial synchronization queue, and ordered synchronization commits are executed through a distributed consensus algorithm. When a cluster anomaly causes synchronization interruption, the operation logs of the master and slave nodes are compared, and the metadata state is verified and repaired according to the dependency order. This invention can ensure the semantic consistency of metadata in a distributed cluster, reduce lock contention during synchronization, improve the concurrent processing capability of metadata operations, and reduce the difficulty of metadata repair in cluster anomaly scenarios.

Eliminating redundant encryption in a distributed file system, including: encrypting, by a first node of a distributed file system, data to be stored in a storage device of the distributed file system; sending, by the first node to a second node of the distributed file system, the encrypted data using an unsecured network connection; sending, by the first node and to the second node of the distributed file system, integrity information for the encrypted data using an authenticated encrypted network connection; and storing, by the second node, the encrypted data into the storage device.

The application discloses a cross-device file operation method, device, medium and product based on an open-source MEC system, and the method comprises the following steps: whenever a standard file operation instruction for a target second device is detected, a target file operation record is created, written into a main storage area, and a record temporary storage area is emptied; the standard file operation instruction is executed through a distributed file system, and after the execution, a corresponding record in the main storage area is updated; when an operation rollback instruction is detected, according to a rollback type of the operation rollback instruction, the latest historical file operation record in the main storage area is transferred to the record temporary storage area, or the latest record in the record temporary storage area is transferred to the main storage area; according to the rollback type and the transferred record, a rollback file operation instruction is determined, and the rollback file operation instruction is executed through the distributed file system, and the technical scheme of the embodiment of the application can significantly improve the operation fault tolerance and user experience of the distributed file management.