580 results about "Distributed File System" patented technology

The invention relates to the technical field of computers and the field of data testing, in particular to a multivariable real-time measurement and state testing method oriented to a data center. The method comprises the following steps: acquiring original data from a sensor, equipment and an application program in a data center, and filtering, cleaning and formatting the original data to generate preprocessed data; storing the preprocessed data to a distributed file system, and constructing a reverse index and B-tree index structure to realize efficient retrieval and redundant backup; intelligent analysis is carried out on the stored data by introducing a machine learning model and an automatic operation and maintenance strategy, the abnormal state of the key performance index is identified, and the resource use condition is predicted; and based on an intelligent analysis result, executing dynamic scheduling and optimal allocation operation of the resources. Through multi-variable real-time monitoring and intelligent scheduling, the resource utilization efficiency and the operation stability of the data center are improved.

The invention relates to the technical field of geographic information systems, in particular to a map tile data efficient access method and system based on hybrid storage and intelligent layering, and the method comprises the following steps: constructing a multi-level storage architecture, carrying out dynamic heat analysis, carrying out intelligent layering scheduling, preloading and space prediction, and carrying out transmission and access optimization. The method has the beneficial effects that the multi-level storage pool is constructed by combining the advantages of object storage and a distributed file system; adopting a dynamic tile popularity analysis model to realize hot data cache acceleration and cold data hierarchical archiving; and meanwhile, a tile request preloading mechanism is introduced, and an access path is optimized in combination with a spatial locality prediction algorithm. According to the method, the storage cost can be remarkably reduced, the tile request delay is reduced, quick response in a high-concurrency scene is supported, and the method is suitable for application scenes such as Internet map services and smart cities.

A method for reducing write latency in a distributed file system. A write request that includes a volume identifier is received at a data management subsystem deployed on a node within a distributed storage system. The data management subsystem maps the volume identifier to a file system volume and maps the file system volume to a set of logical block addresses in a logical block device in a storage management subsystem deployed on the node. The storage management subsystem maps the logical block device to a metadata object for the logical block device on the node that is used to process the write request. The mapping of the file system volume to the set of logical block addresses in the logical block device enables co-locating the metadata object with the file system volume on the node, which reduces the write latency associated with processing the write request.

The invention discloses a data quality detection method and device for DataX and a medium, and relates to the technical field of data processing. The method comprises the following steps: reading a data source in a parallel fragmentation manner through a reader plug-in of DataX to obtain to-be-processed data; when the to-be-processed data is transmitted to the memory buffer area, triggering a quality detection engine, and loading a predefined detection rule base through the quality detection engine; if it is detected that the field is empty, the format deviates from a preset template or the necessary field is missing, generating an abnormal log containing a problem type, a field identifier and original data; and classifying and storing the abnormal logs into a distributed file system according to timestamps, generating a visual abstract report, analyzing problem types, field identifiers and original data, and calling a predefined repair script library for repair. By means of the method, data quality detection is executed in the data input and output process of the DataX, the detection result is output to the text file, and quality detection can be efficiently and accurately conducted on the data.

The invention relates to an intelligent hierarchical access control method and device for railway knowledge and electronic equipment, and the method comprises the steps: registering key certificates for national railway group nodes, railway bureau group nodes and basic-level station nodes, and generating global trusted identity chain certificates; dividing the to-be-accessed railway knowledge assets into four levels, and establishing a relationship between the security level and the user role; loading a railway exclusive strategy expression, and supporting hierarchical inheritance and timeliness logic combination; in response to a user data access request, based on an identity certificate and role attributes, matching a security level of request data with a user role permission threshold, analyzing a railway service dynamic combination condition, judging a current service scene access permission, generating a decryption key or calling an edge gateway to execute outsourcing decryption; and storing the encrypted data to a distributed file system, and recording data hash, a security level and an access operation log. According to the method, the problems of weak identity credibility mechanism, low efficiency of authority management coarse granularity and insufficient industrial characteristic adaptation are solved.

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for securing data. One of the methods includes receiving one or more network data packets. The one or more network data packets include a token that identifies stored sensitive data. The one or more network data packets are desanitized, by: identifying and extracting, from the one or more network data packets, the token; requesting, from a distributed file system, the stored sensitive data, based upon the token; and receiving, in response to the request, the stored sensitive data as received stored sensitive data.

In various examples, data storage is managed using a distributed storage management system that is resilient. Data blocks of a logical block device may be distributed across multiple nodes in a cluster. The logical block device may correspond to a file system volume associated with a file system instance deployed on a selected node within a distributed block layer of a distributed file system. Each data block may have a location in the cluster identified by a block identifier associated with each data block. Each data block may be replicated on at least one other node in the cluster. A metadata object corresponding to a logical block device that maps to the file system volume may be replicated on at least another node in the cluster. Each data block and the metadata object may be hosted on virtualized storage that is protected using redundant array independent disks (RAID).

The invention relates to a distributed file system adaptive to GPU cluster training, a creation method, electronic equipment and a storage medium, and the system comprises a management module which is used for distributing a target GPU cluster to a target training task after the target GPU cluster is distributed to the target training task; a server is created for a plurality of first computer nodes included in the target GPU cluster, a client is created for a plurality of second computer nodes included in the target GPU cluster, and at least one hardware GPU is deployed on each computer node included in the target GPU cluster; the server is used for constructing a GPU cluster distributed file system adaptive to the target training task by utilizing a preset local memory space deployed on the corresponding first computer node; and the client is used for executing IO operation on the GPU cluster distributed file system in the process of executing the target training task by utilizing the target GPU cluster. According to the embodiment of the invention, IO requirements in a GPU cluster training scene can be effectively met.

The invention discloses an automobile production data tracing method, system and device based on a block chain and a medium, and the method comprises the steps: obtaining a product identifier of a production process, original production data and key metadata, storing the original production data to a distributed file system, and generating a unique hash value; packaging the unique hash value and the key metadata to generate a data block, generating block index information according to the unique hash value and the product identifier, and uploading the data block and the block index information to a block chain; obtaining a target hash value and a target data block through indexing on the block chain according to the target product identifier, and determining target metadata according to the target data block; and searching in the distributed file system according to the target hash value to obtain target production data, and generating a target tracing record according to the target production data and the target metadata. The automobile production data tracing efficiency and the accuracy, real-time performance and safety of the automobile production data are improved, and the automobile production data tracing method can be applied to the technical field of automobile production.

The embodiment of the invention discloses an access method in a distributed file system, computing equipment and file access equipment. The method is applied to a first metadata node, the first metadata node belongs to the multiple metadata nodes, the same directory information is stored in each metadata node in the multiple metadata nodes, and the directory information is used for storing directory metadata of the distributed file system. The method comprises the following steps: receiving a first access request from a client, wherein the first access request comprises an object path to be accessed; determining first file fragment information corresponding to first metadata according to the directory information and the to-be-accessed object path; determining an operation result corresponding to the first metadata according to the first file fragment information and the fragment routing information of the first metadata node; and sending the operation result to the client.

The invention relates to the technical field of big data, and discloses a massive small file reading optimization method, system and device and a medium, and the method comprises the following steps: scanning small files in a distributed file system by using a Spark calculation engine to obtain metadata information of the small files; based on the metadata information, small files are classified, and the small files with the same type and data relevance are classified into one group; the resource states of cluster nodes are monitored in real time, the resource states comprise the processor utilization rate, the memory occupancy rate and the I / O load, the load score of each node is calculated, and small file task quotas are dynamically allocated; executing small file merging operation in parallel by utilizing a Spark memory calculation engine to generate a merged file containing a plurality of original small files; creating and maintaining index information in an external database, and recording the position and attribute of each original small file in the combined file; and providing a read-write interface through the special service layer, and positioning and accessing specific small file contents in the combined file.

The invention provides a software copyright transaction and protection method and system based on a block chain, and particularly relates to the technical field of block chain application. The system allows a software supplier to submit software information of different versions, the information is stored in a distributed file system IPFS, and non-tampering storage of copyright information is achieved by indexing hash values of the information in a block chain; in addition, the system adopts an abstract syntax tree to extract a grammar structure of software codes, and copyright protection is further enhanced; the digital signature generated by the copyright information and the copyright information are recorded in a block chain node together, so that the traceability of the copyright information is ensured; when a copyright transaction is generated, the system generates a digital signature containing copyright information, writes the digital signature and transaction information into a block chain node, records each transaction of the copyright, and realizes transparent tracing of the copyright history; in addition, the system further provides a comparison method for suspected software plagiarism, a software author is allowed to upload suspected infringement software, and potential plagiarism behaviors are recognized and warned by comparing abstract syntax tree structures stored on the block chain. According to the method and the system, the efficiency and the safety of software copyright protection are improved, a reliable and transparent platform is provided for software copyright transaction, and reasonable circulation and utilization of the copyright are facilitated.

The invention discloses a distributed file system metadata index optimization method based on remote direct memory access, which comprises directory metadata index based on path name hash: defining a plurality of layers of directories in a directory tree as key directories, and regarding the path as a character string. After calculation of the hash function, determining the storage position of the data in the memory of the metadata server of the distributed file system; and optimizing the memory position of the file metadata based on the popularity, namely placing the file metadata which is frequently accessed at a position adjacent to the key directory metadata to which the file metadata belongs. A metadata management module in a metadata server of the distributed file system regularly checks the popularity of the file and carries out cold and hot migration according to the situation; and client cache optimization oriented to key directory metadata: a metadata cache of a distributed file system client caches the metadata and directory entries of the key directory. Optimally assuming that the metadata and directory entries of the key directory in the cache are not invalid, and updating the cache according to the result of the next-level metadata search. According to the method, the remote reading performance of the metadata index of the distributed file system is optimized by fully utilizing the characteristic of remote direct memory access, the network transmission frequency is reduced, and the load of the metadata server is effectively reduced.

The invention discloses an on-chain and off-chain collaborative interaction system based on a block chain, and the system comprises a data structure module which is used for designing a composite data structure containing metadata, data content, a hash value and a timestamp, and employs a Merkle tree to manage data; the distributed storage and transmission module is used for storing data under the chain by adopting a distributed file system, storing key data on the chain by using a block chain, and designing a secure transmission protocol; by means of the Merkle tree and Hash verification, data integrity can be quickly and accurately verified, tampering can be perceived, a distributed storage and transmission technology and a two-way synchronization strategy can be designed, on-chain and off-chain efficient collaboration is achieved, data processing efficiency and reliability are improved, data security privacy is guaranteed by means of block chain characteristics and encryption, digital signature and other technologies, and the method is suitable for large-scale popularization and application. Distributed storage and a P2P network facilitate node and capability expansion to adapt to data growth, an intelligent contract is introduced, online and offline data collaborative automatic management is achieved, risks are reduced, and efficiency is improved.

The invention provides a Hadoop cluster incremental backup recovery method based on data block optimization, and relates to the technical field of data management, and the method comprises the following steps: obtaining distribution state information and historical change records of data blocks in a distributed file system, extracting space-time correlation characteristics, and constructing a data block dependency graph containing dependency direction and dependency intensity; identifying the changed data blocks, performing topological sorting, and identifying key path data blocks and independent data block clusters; calculating a distribution entropy value according to a copy distribution position, and determining a backup granularity type; a backup execution sequence is generated according to a topological sorting result, key path data blocks are backed up preferentially, independent data block clusters are backed up in parallel, and backup metadata containing dependency relationship identifiers and distribution entropy values are generated; and performing dependency consistency verification after recovery based on the dependency relationship identifier. According to the method, the backup recovery efficiency is improved, the data consistency is guaranteed, and the storage overhead is reduced.

The invention provides a file fuzzy copying method and system based on a big data file cluster, and relates to the technical field of file copying, the file fuzzy copying method comprises the steps that file content, file names and metadata feature vectors are extracted from a to-be-matched file set, and the file content feature vectors are obtained based on deep learning model coding; afterwards, a distributed computing framework is used for parallel computing of similarity scores of the to-be-matched files and the files in the target file set, the scores are obtained through weighted computing of file content, file names and metadata feature vector similarity, and a to-be-copied file list is generated through screening according to a preset threshold value. And finally, the distributed file system dynamically allocates a replication task according to system resources, executes file replication based on a data block level parallel transmission mechanism, verifies data integrity and generates a replication task execution report. According to the method, fuzzy file copying can be efficiently and accurately carried out in the big data file cluster, the file copying efficiency and accuracy are improved, and the system resource consumption is reduced.

The invention discloses a distributed caching method and system and an acceleration method for large model reasoning service. The system comprises a confidential virtual machine cluster, a zero-trust distributed caching system and a FUSE client. The zero-trust distributed cache system comprises a data strategy service, a data engine plug-in, a distributed file system and a trusted access service. And the FUSE client is deployed at the confidential container end, and establishes secure connection with the trusted access service based on the remote proof and the identity legality confirmation proof of the corresponding confidential container so as to provide a plaintext data access service for the corresponding confidential container. According to the scheme, a distributed cache implementation scheme is provided for the confidential computing container cluster, and the problem of performance loss caused by repeatedly reading, encrypting and decrypting the same data from an external storage system is avoided while the data security is guaranteed by using the large-scale confidential container cluster; and the storage access efficiency in scenes such as model training and reasoning is improved.

The invention discloses an intelligent elevator information management system based on computer vision. The intelligent elevator information management system comprises a data sensing module, a vision detection module, a strategy network construction module and an information management module. The invention relates to the technical field of elevator intelligent management, in particular to an intelligent elevator information management system based on computer vision, which realizes passenger tracking and accurate counting by adopting a time-space spectrum threshold value guided by optical flow to perform optimal cutting and fusing visual detection confidence and Kalman filtering posterior probability; double attention and multi-scale residual error updating are introduced, and short-time passenger flow high-precision prediction is achieved; based on incremental hierarchical clustering, Poisson load estimation and a strategy network, elevator scheduling is dynamically optimized, and waiting time and energy consumption are reduced; and a relational database and distributed file system mixed storage architecture is constructed through a real-time data display single information management aspect, so that the data consistency, the query access efficiency and the decision support capability are enhanced.

The invention provides a satellite grouping management system for a satellite cloud computing internet platform. Comprising three modules: a grouping management module, a task scheduling module and a low-delay service migration management module, and the three modules all run on a management node k3s master of k3s; in the functional architecture, the client side skclient accesses an API provided by a system through a remote procedure call frame gRPC, and is used for submitting tasks to a task scheduling module and a low-delay service migration management module in a management node k3s master and adding and deleting a peer relation to a grouping management module; the API submits a task or service to the skagent on the working node satellite, and after the access of a user reaches the skagent running on the working node, the skagent running on the working node forwards a user request to a corresponding module of a management node k3s master of k3s; and the skagent running on each working node is also in communication connection with a satellite distributed file system. The problem that a resource scheduling system adapts to a satellite network with wide geographical distribution and high dynamic time-varying characteristics is solved.

The invention discloses a single-file concurrent read-write method and system of a distributed file system. The method comprises the steps that S1, a conservative write range lock application request of a preset interval is sent to a metadata server; the range lock takes a block as the minimum granularity; s2, segmenting to-be-written data into a plurality of data blocks according to a preset block length, and applying for a corresponding locking interval in a local memory lock table according to a starting position and an ending position of each data block in a target file; judging whether the currently applied locking interval is overlapped with the registered locking interval or not; s3, if the data block is not in the write range lock, applying for an interval for expanding the write range lock from the metadata server; s4, distributing the segmented data block write-in requests to block engines of corresponding storage nodes, so that the block engines add data block mutual exclusion locks at corresponding positions; and S5, in response to a request of closing or refreshing the target file by the user, sending a request of releasing the write range lock interval. According to the method and the device, the performance is not reduced on the premise of improving the strong consistency.

The invention provides a request processing method and system, and relates to the field of data storage, and the method comprises the steps: in response to a data operation request, determining a target physical memory address applied by target data needing to be accessed by the data operation request in a graphics processor; establishing a transmission path between the distributed file system and the graphics processor based on the target physical memory address; segmenting the target data into a plurality of data segments with different data attributes; selecting a corresponding target transmission protocol for the plurality of data segments, and distributing the plurality of data segments to different transmission sub-paths; determining a target position of the target data in the distributed file system according to the data operation request; and transmitting the plurality of data segments from a target position in the distributed file system to a target physical memory address of the graphics processor in parallel. According to the customized transmission mode, low efficiency and delay of a one-cutting mode are avoided, and the transmission efficiency is improved.

The invention discloses a blockchain data security storage method based on verifiable secret sharing. The method comprises the following steps executed by computer hardware: receiving to-be-stored data, performing encryption processing on the to-be-stored data to obtain a ciphertext, storing the ciphertext to a distributed file system, obtaining a storage address, and calculating a hash value of the ciphertext; dividing the encryption key into a plurality of sub-key fragments according to set parameters, and calculating a hash value of each sub-key fragment; packaging the sub-key fragment, the sub-key fragment hash value and the ciphertext hash value into a transaction unit, and writing the transaction unit into the block chain; receiving a decryption request, and reconstructing an encryption key when the number of submitted effective sub-key fragments reaches a threshold value; and acquiring the ciphertext from the distributed file system, verifying the hash value of the ciphertext, and decrypting by using the reconstructed encryption key to obtain the plaintext. According to the invention, the security management and control of the full life cycle of the data are realized, the integrity, confidentiality and authenticability of the data and the secret key are ensured, and the security of the data in the processes of storage, transmission, secret key management and the like is ensured.

The invention discloses a spatio-temporal data hybrid storage system and device based on a smart city and a storage medium, and relates to the technical field of data processing, the spatio-temporal data hybrid storage system based on the smart city comprises a processing module and a storage module, and the storage module comprises a file system and a database cluster. Data interaction is carried out between the file system and the database cluster, the file system comprises a local file system and a distributed file system, the database cluster comprises a relational database, a non-relational database and a search database, and data interaction is carried out between the relational database and the non-relational database. And data interaction is performed between the non-relational database and the search database. The problems that spatio-temporal data storage and management are disordered and are not easy to manage, and spatio-temporal data read-write analysis and calculation efficiency in the field of smart cities is low are solved, and the use performance and efficiency of the spatio-temporal data in the field of smart cities are improved.

The invention relates to a distributed file system based on a data processing unit (DPU), which comprises a computing node, the DPU and a storage node, wherein the computing node comprises a client request interface and a host shared memory; the DPU comprises a file request processing module, a data path management module, a metadata index management module and a metadata memory management module; the storage node comprises a far-end shared memory; the file request processing module adopts a unilateral polling mechanism; the data path management module adopts single-operation zero-copy transmission; the metadata index management module adopts a core index structure; and the metadata memory management module adopts a double-layer distribution structure. According to the method, a unilateral polling mechanism is adopted in the DPU, the CPU overhead of the computing nodes is reduced according to a core index structure and a double-layer distribution structure, and meanwhile, an efficient data path is constructed for breaking through the memory bottleneck of the DPU.

This disclosure presents a system for rectifying server failures in distributed file systems using predictive logical markers. The system begins by receiving user details, a file name, and a block address via a distributed file system (DFS) client. If previous read information exists for the given file and user, a trained machine learning (ML) model predicts logical markers for file fragments. The ML model then transmits remaining block addresses to the DFS client. Concurrently, the system facilitates a data input stream, communicating predicted block addresses between the DFS client and the ML model. Block location information is retrieved from a distributed ledger via a primary node. The data input stream is processed via a secondary node, allowing efficient rectification of server failures.

The invention discloses a Kubernetes-based multi-tenant resource dynamic isolation method, which comprises the following steps of: constructing a CPU (Central Processing Unit) resource control group by utilizing cgropsv2, and realizing dynamic allocation of CPU resources in combination with a dynamic binding engine; constructing a physically isolated storage pool based on the Ceph distributed file system; the network namespace and eBPF flow control technology is utilized, and LWTunnel packaging is combined to realize tenant network complete isolation and QoS speed limitation; constructing an LSTM neural network prediction model to pre-judge a resource demand; a multi-level security system is constructed through AES-256 encryption, Seccomp / BPF container limitation and other technologies; and defining a dynamic strategy object by utilizing CRD to realize strategy management. Through dynamic adaptive resource scheduling, multi-level security protection and highly flexible customization capability, the cluster resource utilization rate can be improved, the CPU cross-NUMA access delay can be reduced, the malicious traffic interception rate can be improved, and the high-performance and high-security requirements of multi-industry and multi-tenant scenes such as finance and e-commerce can be met.

The invention provides a data processing method and system, electronic equipment, a storage medium and a product, and relates to the technical field of information. The data processing method comprises the following steps: collecting original reconciliation data of each business department, taking a distributed file system as bottom storage, constructing a data warehouse framework and a distributed column-oriented open source database on the basis, and managing the original reconciliation data. Based on a user query instruction, target account checking data of a current checking time period is obtained from an open source database, and the data relates to more than two business departments. And then target reconciliation data are verified, and data of different departments are checked in a cross manner. According to the scheme, a large amount of data is efficiently stored through the distributed file system, ordered management and efficient analysis are achieved through an upper-layer framework and a database, quick and accurate response query can be achieved, the data accuracy and reliability are improved, potential business problems can be perceived in time, and business promotion and financial accounting accuracy are guaranteed.

The invention discloses a trusted data tracing system and method for a prefabricated dish supply chain. The system comprises an edge data acquisition module, an equipment authentication module, an off-chain storage module, an on-chain evidence storage module, a privacy verification module and an intelligent contract execution module. And the edge data acquisition module obtains state data of the prefabricated dishes at the service node and generates an event log. And the equipment authentication module distributes an equipment identity label and performs digital signature on the event log. And the under-chain storage module encrypts the signature log, stores the encrypted signature log into a distributed file system, and generates a content identifier. And the on-chain evidence storage module writes the content identifier, the device identity identifier, the timestamp and the event type code into the alliance block chain. And the privacy verification module responds to the verification request, extracts numerical parameters, generates zero-knowledge proof located in a preset interval, and submits the zero-knowledge proof to the alliance block chain for verification. And the intelligent contract execution module executes business rule logic, and triggers state judgment or abnormal marking based on an on-chain event.

The invention relates to the field of data storage, and provides a crop phenotype data storage and retrieval method and device, equipment and a storage medium, and the method comprises the steps: obtaining multi-modal data comprising crop phenotype data and metadata information; storing the structured data and the metadata information to a distributed database, and storing the unstructured data to a distributed file system; synchronously storing a retrieval field of the structured data, a retrieval field of the small file data and metadata information of the large file data into a distributed search analysis engine; and under the condition that the data matched with the user retrieval condition does not exist in the memory database, determining a retrieval result based on the user retrieval condition and the distributed search analysis engine. According to the method, the multi-modal crop phenotype data are classified and stored according to different data types, so that the storage efficiency and the retrieval rate of the multi-modal crop phenotype data are improved.

The embodiment of the invention aims to provide a method and equipment for storing data in a Hadoop distributed file system, a computer program product and a computer program storage medium. Wherein the table partitions are divided into cold data, temperature data or hot data according to the data access frequencies of the table partitions, and the data access frequencies of the cold data, the temperature data and the hot data are sequentially increased; according to the classification of each table partition, corresponding storage strategies are executed, and the storage efficiency of the cold data, the storage efficiency of the temperature data and the storage efficiency of the hot data are sequentially reduced. According to the embodiments of the invention, the system can provide corresponding storage strategies according to data with different temperatures, which is accurate to a table partition level, so that smooth migration of the data to EC storage is realized, normal access of services is not affected in the process, and high efficiency and availability of the system are ensured.