856 results about "Distributed File System" patented technology

The invention relates to the technical field of data management and the field of financial science and technology, and discloses a data management method based on cold data migration, which comprises the following steps: extracting metadata in a distributed file system, and importing the metadata into a metadata storage library of a distributed data processing platform. And analyzing file information in the metadata storage library through an offline analysis task, identifying cold data, and storing file path information of the cold data in a database. And according to the identified cold data, generating a data migration plan comprising a migration target position, migration priority ranking and migration time arrangement. And migrating the cold data from the original storage position to the target far-end storage, and updating a table partition path in the metadata storage library to point to a new storage position. And after migration is completed, deleting the cold data in the original storage position. According to the method, the cold data and the computing resources are effectively separated, the utilization efficiency of the storage resources is optimized, the overall resource consumption of the system is reduced, and meanwhile the expansibility and the performance of the system are improved.

The invention relates to the technical field of computers and the field of data testing, in particular to a multivariable real-time measurement and state testing method oriented to a data center. The method comprises the following steps: acquiring original data from a sensor, equipment and an application program in a data center, and filtering, cleaning and formatting the original data to generate preprocessed data; storing the preprocessed data to a distributed file system, and constructing a reverse index and B-tree index structure to realize efficient retrieval and redundant backup; intelligent analysis is carried out on the stored data by introducing a machine learning model and an automatic operation and maintenance strategy, the abnormal state of the key performance index is identified, and the resource use condition is predicted; and based on an intelligent analysis result, executing dynamic scheduling and optimal allocation operation of the resources. Through multi-variable real-time monitoring and intelligent scheduling, the resource utilization efficiency and the operation stability of the data center are improved.

Embodiments are directed to managing data in a file system. In response to a request from a client of a spoke file system to access file system items shared by a hub file system: determining a portion of the spoke file systems that may have an existing lock on the file system items; obtaining log entries from the portion of the spoke file systems such that the log entries are associated with commands that were executed in the portion of the spoke file systems; executing the commands in the hub file system to update the file system items in the hub file system; granting a lock on the file system items in the hub file system that enables the client to access the updated file system items from the spoke file systems.

The invention relates to the technical field of geographic information systems, in particular to a map tile data efficient access method and system based on hybrid storage and intelligent layering, and the method comprises the following steps: constructing a multi-level storage architecture, carrying out dynamic heat analysis, carrying out intelligent layering scheduling, preloading and space prediction, and carrying out transmission and access optimization. The method has the beneficial effects that the multi-level storage pool is constructed by combining the advantages of object storage and a distributed file system; adopting a dynamic tile popularity analysis model to realize hot data cache acceleration and cold data hierarchical archiving; and meanwhile, a tile request preloading mechanism is introduced, and an access path is optimized in combination with a spatial locality prediction algorithm. According to the method, the storage cost can be remarkably reduced, the tile request delay is reduced, quick response in a high-concurrency scene is supported, and the method is suitable for application scenes such as Internet map services and smart cities.

A distributed storage management system comprising nodes that form a cluster, a distributed block layer that spans the nodes in the cluster, and file system instances deployed on the nodes. Each file system instance comprises a data management subsystem and a storage management subsystem disaggregated from the data management subsystem. The storage management subsystem comprises a node block store that forms a portion of the distributed block layer and a storage manager that manages a key-value store and virtualized storage supporting the node block store. A file system volume hosted by the data management subsystem maps to a logical block device hosted by the virtualized storage in the storage management subsystem. The key-value store includes, for a data block of the logical block device, a key that comprises a block identifier for the logical block device and a value that comprises the data block.

A method for reducing write latency in a distributed file system. A write request that includes a volume identifier is received at a data management subsystem deployed on a node within a distributed storage system. The data management subsystem maps the volume identifier to a file system volume and maps the file system volume to a set of logical block addresses in a logical block device in a storage management subsystem deployed on the node. The storage management subsystem maps the logical block device to a metadata object for the logical block device on the node that is used to process the write request. The mapping of the file system volume to the set of logical block addresses in the logical block device enables co-locating the metadata object with the file system volume on the node, which reduces the write latency associated with processing the write request.

The invention discloses a data quality detection method and device for DataX and a medium, and relates to the technical field of data processing. The method comprises the following steps: reading a data source in a parallel fragmentation manner through a reader plug-in of DataX to obtain to-be-processed data; when the to-be-processed data is transmitted to the memory buffer area, triggering a quality detection engine, and loading a predefined detection rule base through the quality detection engine; if it is detected that the field is empty, the format deviates from a preset template or the necessary field is missing, generating an abnormal log containing a problem type, a field identifier and original data; and classifying and storing the abnormal logs into a distributed file system according to timestamps, generating a visual abstract report, analyzing problem types, field identifiers and original data, and calling a predefined repair script library for repair. By means of the method, data quality detection is executed in the data input and output process of the DataX, the detection result is output to the text file, and quality detection can be efficiently and accurately conducted on the data.

The invention relates to an intelligent hierarchical access control method and device for railway knowledge and electronic equipment, and the method comprises the steps: registering key certificates for national railway group nodes, railway bureau group nodes and basic-level station nodes, and generating global trusted identity chain certificates; dividing the to-be-accessed railway knowledge assets into four levels, and establishing a relationship between the security level and the user role; loading a railway exclusive strategy expression, and supporting hierarchical inheritance and timeliness logic combination; in response to a user data access request, based on an identity certificate and role attributes, matching a security level of request data with a user role permission threshold, analyzing a railway service dynamic combination condition, judging a current service scene access permission, generating a decryption key or calling an edge gateway to execute outsourcing decryption; and storing the encrypted data to a distributed file system, and recording data hash, a security level and an access operation log. According to the method, the problems of weak identity credibility mechanism, low efficiency of authority management coarse granularity and insufficient industrial characteristic adaptation are solved.

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for securing data. One of the methods includes receiving one or more network data packets. The one or more network data packets include a token that identifies stored sensitive data. The one or more network data packets are desanitized, by: identifying and extracting, from the one or more network data packets, the token; requesting, from a distributed file system, the stored sensitive data, based upon the token; and receiving, in response to the request, the stored sensitive data as received stored sensitive data.

The invention relates to the field of intelligent fault diagnosis, in particular to a system and method for integrating intelligent fault diagnosis of an oiling machine, and the method comprises the following steps: S1, integrating operation data obtained by sensors of an oil way, a circuit and associated equipment of the oiling machine in real time, and constructing a distributed framework based on a Spark core component and a Hadoop distributed file system; data storage is carried out through an HDFS; s2, continuously collected operation data of the oiling machine are input into the fault diagnosis model, and real-time anomaly detection is carried out; s3, analyzing the operation data through the fault prediction model, and predicting the fault probability of the oiling machine in the long-term change trend or the short-term change trend of the operation data; and S4, performing graded early warning according to the detected or predicted fault condition. According to the invention, the fault detection accuracy can be improved, real-time monitoring and early warning are realized, diagnosis and maintenance are facilitated, and the operation management level of the gas station is improved.

In various examples, data storage is managed using a distributed storage management system that is resilient. Data blocks of a logical block device may be distributed across multiple nodes in a cluster. The logical block device may correspond to a file system volume associated with a file system instance deployed on a selected node within a distributed block layer of a distributed file system. Each data block may have a location in the cluster identified by a block identifier associated with each data block. Each data block may be replicated on at least one other node in the cluster. A metadata object corresponding to a logical block device that maps to the file system volume may be replicated on at least another node in the cluster. Each data block and the metadata object may be hosted on virtualized storage that is protected using redundant array independent disks (RAID).

The invention discloses an industrial equipment evidence obtaining data management method and system, equipment and a medium, and relates to the field of industrial data safety management. The method comprises the following steps: acquiring different types of evidence obtaining data of industrial equipment; carrying out cleaning and attribute labeling on the evidence obtaining data to obtain pre-processed evidence obtaining data; generating a unique hash value for the pre-processed evidence obtaining data, and writing the hash value and the labeled attribute into the block chain; storing the processed evidence obtaining data into a distributed file system and a database; performing multi-signature access control based on the smart contract of the block chain, and allowing a user to access requested data from a distributed file system and / or a database after the access request is approved; and recording each data access operation in the block chain through the smart contract. According to the method, high-reliability guarantee of the whole process of collecting, storing and accessing the evidence obtaining data of the industrial equipment is realized.

The invention relates to a distributed file system adaptive to GPU cluster training, a creation method, electronic equipment and a storage medium, and the system comprises a management module which is used for distributing a target GPU cluster to a target training task after the target GPU cluster is distributed to the target training task; a server is created for a plurality of first computer nodes included in the target GPU cluster, a client is created for a plurality of second computer nodes included in the target GPU cluster, and at least one hardware GPU is deployed on each computer node included in the target GPU cluster; the server is used for constructing a GPU cluster distributed file system adaptive to the target training task by utilizing a preset local memory space deployed on the corresponding first computer node; and the client is used for executing IO operation on the GPU cluster distributed file system in the process of executing the target training task by utilizing the target GPU cluster. According to the embodiment of the invention, IO requirements in a GPU cluster training scene can be effectively met.

The invention discloses an automobile production data tracing method, system and device based on a block chain and a medium, and the method comprises the steps: obtaining a product identifier of a production process, original production data and key metadata, storing the original production data to a distributed file system, and generating a unique hash value; packaging the unique hash value and the key metadata to generate a data block, generating block index information according to the unique hash value and the product identifier, and uploading the data block and the block index information to a block chain; obtaining a target hash value and a target data block through indexing on the block chain according to the target product identifier, and determining target metadata according to the target data block; and searching in the distributed file system according to the target hash value to obtain target production data, and generating a target tracing record according to the target production data and the target metadata. The automobile production data tracing efficiency and the accuracy, real-time performance and safety of the automobile production data are improved, and the automobile production data tracing method can be applied to the technical field of automobile production.

The embodiment of the invention discloses an access method in a distributed file system, computing equipment and file access equipment. The method is applied to a first metadata node, the first metadata node belongs to the multiple metadata nodes, the same directory information is stored in each metadata node in the multiple metadata nodes, and the directory information is used for storing directory metadata of the distributed file system. The method comprises the following steps: receiving a first access request from a client, wherein the first access request comprises an object path to be accessed; determining first file fragment information corresponding to first metadata according to the directory information and the to-be-accessed object path; determining an operation result corresponding to the first metadata according to the first file fragment information and the fragment routing information of the first metadata node; and sending the operation result to the client.

The invention relates to a memory cache framework system for a distributed file system client, which comprises a memory pool layer used for allocating a first memory space in a memory pool in response to a first request; the memory pool is divided into an idle list management memory space and a standby memory space; the data segment layer is used for linking a memory address corresponding to the file data cached in the memory pool through a double linked list to obtain a logic address corresponding to the file data; the file layer is used for responding to a file reading request, realizing a reading operation on a specified reading data segment by adding a reading lock to the specified reading data segment, and responding to a file writing request, realizing a writing operation on a specified writing data segment by adding a writing lock to the specified writing data segment; the monitoring layer is used for managing the memory use condition of the memory pool and monitoring the state of the data segment; according to the memory cache framework system, the data cache performance can be remarkably improved, the memory use pressure is reduced, and the data consistency and security are ensured.

The invention discloses a BIM data efficient storage and access method based on a distributed architecture, and the method comprises the following steps: S1, data distributed storage design: employing a highly extensible and fault-tolerant distributed file system as a basis, and distributing BIM data to a plurality of storage nodes; the system resource consumption is reduced; the user experience is improved; s3, establishing a multi-level and multi-dimensional data index system so as to quickly and accurately retrieve the BIM data; and S4, optimizing data synchronization and consistency to meet the requirements of high real-time performance and high reliability of the BIM data. Through the design of the distributed architecture, the BIM data are stored on a plurality of nodes in a scattered manner, and the storage capacity and the processing capacity of the data are remarkably improved. Through data fragmentation and load balancing technologies, the problems are effectively relieved, so that the system can efficiently process massive BIM data.

The invention relates to the technical field of big data, and discloses a massive small file reading optimization method, system and device and a medium, and the method comprises the following steps: scanning small files in a distributed file system by using a Spark calculation engine to obtain metadata information of the small files; based on the metadata information, small files are classified, and the small files with the same type and data relevance are classified into one group; the resource states of cluster nodes are monitored in real time, the resource states comprise the processor utilization rate, the memory occupancy rate and the I / O load, the load score of each node is calculated, and small file task quotas are dynamically allocated; executing small file merging operation in parallel by utilizing a Spark memory calculation engine to generate a merged file containing a plurality of original small files; creating and maintaining index information in an external database, and recording the position and attribute of each original small file in the combined file; and providing a read-write interface through the special service layer, and positioning and accessing specific small file contents in the combined file.

The invention provides a software copyright transaction and protection method and system based on a block chain, and particularly relates to the technical field of block chain application. The system allows a software supplier to submit software information of different versions, the information is stored in a distributed file system IPFS, and non-tampering storage of copyright information is achieved by indexing hash values of the information in a block chain; in addition, the system adopts an abstract syntax tree to extract a grammar structure of software codes, and copyright protection is further enhanced; the digital signature generated by the copyright information and the copyright information are recorded in a block chain node together, so that the traceability of the copyright information is ensured; when a copyright transaction is generated, the system generates a digital signature containing copyright information, writes the digital signature and transaction information into a block chain node, records each transaction of the copyright, and realizes transparent tracing of the copyright history; in addition, the system further provides a comparison method for suspected software plagiarism, a software author is allowed to upload suspected infringement software, and potential plagiarism behaviors are recognized and warned by comparing abstract syntax tree structures stored on the block chain. According to the method and the system, the efficiency and the safety of software copyright protection are improved, a reliable and transparent platform is provided for software copyright transaction, and reasonable circulation and utilization of the copyright are facilitated.

The invention discloses a distributed file system metadata index optimization method based on remote direct memory access, which comprises directory metadata index based on path name hash: defining a plurality of layers of directories in a directory tree as key directories, and regarding the path as a character string. After calculation of the hash function, determining the storage position of the data in the memory of the metadata server of the distributed file system; and optimizing the memory position of the file metadata based on the popularity, namely placing the file metadata which is frequently accessed at a position adjacent to the key directory metadata to which the file metadata belongs. A metadata management module in a metadata server of the distributed file system regularly checks the popularity of the file and carries out cold and hot migration according to the situation; and client cache optimization oriented to key directory metadata: a metadata cache of a distributed file system client caches the metadata and directory entries of the key directory. Optimally assuming that the metadata and directory entries of the key directory in the cache are not invalid, and updating the cache according to the result of the next-level metadata search. According to the method, the remote reading performance of the metadata index of the distributed file system is optimized by fully utilizing the characteristic of remote direct memory access, the network transmission frequency is reduced, and the load of the metadata server is effectively reduced.

The invention discloses an on-chain and off-chain collaborative interaction system based on a block chain, and the system comprises a data structure module which is used for designing a composite data structure containing metadata, data content, a hash value and a timestamp, and employs a Merkle tree to manage data; the distributed storage and transmission module is used for storing data under the chain by adopting a distributed file system, storing key data on the chain by using a block chain, and designing a secure transmission protocol; by means of the Merkle tree and Hash verification, data integrity can be quickly and accurately verified, tampering can be perceived, a distributed storage and transmission technology and a two-way synchronization strategy can be designed, on-chain and off-chain efficient collaboration is achieved, data processing efficiency and reliability are improved, data security privacy is guaranteed by means of block chain characteristics and encryption, digital signature and other technologies, and the method is suitable for large-scale popularization and application. Distributed storage and a P2P network facilitate node and capability expansion to adapt to data growth, an intelligent contract is introduced, online and offline data collaborative automatic management is achieved, risks are reduced, and efficiency is improved.

The invention provides a Hadoop cluster incremental backup recovery method based on data block optimization, and relates to the technical field of data management, and the method comprises the following steps: obtaining distribution state information and historical change records of data blocks in a distributed file system, extracting space-time correlation characteristics, and constructing a data block dependency graph containing dependency direction and dependency intensity; identifying the changed data blocks, performing topological sorting, and identifying key path data blocks and independent data block clusters; calculating a distribution entropy value according to a copy distribution position, and determining a backup granularity type; a backup execution sequence is generated according to a topological sorting result, key path data blocks are backed up preferentially, independent data block clusters are backed up in parallel, and backup metadata containing dependency relationship identifiers and distribution entropy values are generated; and performing dependency consistency verification after recovery based on the dependency relationship identifier. According to the method, the backup recovery efficiency is improved, the data consistency is guaranteed, and the storage overhead is reduced.

Methods, systems, and devices for staging of non-fungible tokens (NFTs) before deployment of the NFTs are described. A custodial token platform receives user inputs corresponding to a collection of NFTs associated with a smart contract to be stored on a blockchain distributed data store. The custodial token platform deploys the smart contract to the blockchain distributed data store based on the user inputs, receives data corresponding to the collection, and uploads the data corresponding to the collection to a distributed file system. Uploading the data corresponding to the collection includes uploading first content for a first NFT and uploading second content for a second NFT to the distributed file system. The custodial token platform receives an update to the first content, uploads the updated first content to the distributed file system, and updates the smart contract to include indications of the data corresponding to the collection and the updated first content.

The invention provides a file fuzzy copying method and system based on a big data file cluster, and relates to the technical field of file copying, the file fuzzy copying method comprises the steps that file content, file names and metadata feature vectors are extracted from a to-be-matched file set, and the file content feature vectors are obtained based on deep learning model coding; afterwards, a distributed computing framework is used for parallel computing of similarity scores of the to-be-matched files and the files in the target file set, the scores are obtained through weighted computing of file content, file names and metadata feature vector similarity, and a to-be-copied file list is generated through screening according to a preset threshold value. And finally, the distributed file system dynamically allocates a replication task according to system resources, executes file replication based on a data block level parallel transmission mechanism, verifies data integrity and generates a replication task execution report. According to the method, fuzzy file copying can be efficiently and accurately carried out in the big data file cluster, the file copying efficiency and accuracy are improved, and the system resource consumption is reduced.

The invention discloses a distributed caching method and system and an acceleration method for large model reasoning service. The system comprises a confidential virtual machine cluster, a zero-trust distributed caching system and a FUSE client. The zero-trust distributed cache system comprises a data strategy service, a data engine plug-in, a distributed file system and a trusted access service. And the FUSE client is deployed at the confidential container end, and establishes secure connection with the trusted access service based on the remote proof and the identity legality confirmation proof of the corresponding confidential container so as to provide a plaintext data access service for the corresponding confidential container. According to the scheme, a distributed cache implementation scheme is provided for the confidential computing container cluster, and the problem of performance loss caused by repeatedly reading, encrypting and decrypting the same data from an external storage system is avoided while the data security is guaranteed by using the large-scale confidential container cluster; and the storage access efficiency in scenes such as model training and reasoning is improved.

The invention discloses an intelligent elevator information management system based on computer vision. The intelligent elevator information management system comprises a data sensing module, a vision detection module, a strategy network construction module and an information management module. The invention relates to the technical field of elevator intelligent management, in particular to an intelligent elevator information management system based on computer vision, which realizes passenger tracking and accurate counting by adopting a time-space spectrum threshold value guided by optical flow to perform optimal cutting and fusing visual detection confidence and Kalman filtering posterior probability; double attention and multi-scale residual error updating are introduced, and short-time passenger flow high-precision prediction is achieved; based on incremental hierarchical clustering, Poisson load estimation and a strategy network, elevator scheduling is dynamically optimized, and waiting time and energy consumption are reduced; and a relational database and distributed file system mixed storage architecture is constructed through a real-time data display single information management aspect, so that the data consistency, the query access efficiency and the decision support capability are enhanced.

The invention provides a satellite grouping management system for a satellite cloud computing internet platform. Comprising three modules: a grouping management module, a task scheduling module and a low-delay service migration management module, and the three modules all run on a management node k3s master of k3s; in the functional architecture, the client side skclient accesses an API provided by a system through a remote procedure call frame gRPC, and is used for submitting tasks to a task scheduling module and a low-delay service migration management module in a management node k3s master and adding and deleting a peer relation to a grouping management module; the API submits a task or service to the skagent on the working node satellite, and after the access of a user reaches the skagent running on the working node, the skagent running on the working node forwards a user request to a corresponding module of a management node k3s master of k3s; and the skagent running on each working node is also in communication connection with a satellite distributed file system. The problem that a resource scheduling system adapts to a satellite network with wide geographical distribution and high dynamic time-varying characteristics is solved.

The invention discloses a single-file concurrent read-write method and system of a distributed file system. The method comprises the steps that S1, a conservative write range lock application request of a preset interval is sent to a metadata server; the range lock takes a block as the minimum granularity; s2, segmenting to-be-written data into a plurality of data blocks according to a preset block length, and applying for a corresponding locking interval in a local memory lock table according to a starting position and an ending position of each data block in a target file; judging whether the currently applied locking interval is overlapped with the registered locking interval or not; s3, if the data block is not in the write range lock, applying for an interval for expanding the write range lock from the metadata server; s4, distributing the segmented data block write-in requests to block engines of corresponding storage nodes, so that the block engines add data block mutual exclusion locks at corresponding positions; and S5, in response to a request of closing or refreshing the target file by the user, sending a request of releasing the write range lock interval. According to the method and the device, the performance is not reduced on the premise of improving the strong consistency.

The invention provides a request processing method and system, and relates to the field of data storage, and the method comprises the steps: in response to a data operation request, determining a target physical memory address applied by target data needing to be accessed by the data operation request in a graphics processor; establishing a transmission path between the distributed file system and the graphics processor based on the target physical memory address; segmenting the target data into a plurality of data segments with different data attributes; selecting a corresponding target transmission protocol for the plurality of data segments, and distributing the plurality of data segments to different transmission sub-paths; determining a target position of the target data in the distributed file system according to the data operation request; and transmitting the plurality of data segments from a target position in the distributed file system to a target physical memory address of the graphics processor in parallel. According to the customized transmission mode, low efficiency and delay of a one-cutting mode are avoided, and the transmission efficiency is improved.

The invention discloses a blockchain data security storage method based on verifiable secret sharing. The method comprises the following steps executed by computer hardware: receiving to-be-stored data, performing encryption processing on the to-be-stored data to obtain a ciphertext, storing the ciphertext to a distributed file system, obtaining a storage address, and calculating a hash value of the ciphertext; dividing the encryption key into a plurality of sub-key fragments according to set parameters, and calculating a hash value of each sub-key fragment; packaging the sub-key fragment, the sub-key fragment hash value and the ciphertext hash value into a transaction unit, and writing the transaction unit into the block chain; receiving a decryption request, and reconstructing an encryption key when the number of submitted effective sub-key fragments reaches a threshold value; and acquiring the ciphertext from the distributed file system, verifying the hash value of the ciphertext, and decrypting by using the reconstructed encryption key to obtain the plaintext. According to the invention, the security management and control of the full life cycle of the data are realized, the integrity, confidentiality and authenticability of the data and the secret key are ensured, and the security of the data in the processes of storage, transmission, secret key management and the like is ensured.