443 results about "Distributed File System" patented technology

The invention relates to the technical field of geographic information systems, in particular to a map tile data efficient access method and system based on hybrid storage and intelligent layering, and the method comprises the following steps: constructing a multi-level storage architecture, carrying out dynamic heat analysis, carrying out intelligent layering scheduling, preloading and space prediction, and carrying out transmission and access optimization. The method has the beneficial effects that the multi-level storage pool is constructed by combining the advantages of object storage and a distributed file system; adopting a dynamic tile popularity analysis model to realize hot data cache acceleration and cold data hierarchical archiving; and meanwhile, a tile request preloading mechanism is introduced, and an access path is optimized in combination with a spatial locality prediction algorithm. According to the method, the storage cost can be remarkably reduced, the tile request delay is reduced, quick response in a high-concurrency scene is supported, and the method is suitable for application scenes such as Internet map services and smart cities.

The invention discloses a data quality detection method and device for DataX and a medium, and relates to the technical field of data processing. The method comprises the following steps: reading a data source in a parallel fragmentation manner through a reader plug-in of DataX to obtain to-be-processed data; when the to-be-processed data is transmitted to the memory buffer area, triggering a quality detection engine, and loading a predefined detection rule base through the quality detection engine; if it is detected that the field is empty, the format deviates from a preset template or the necessary field is missing, generating an abnormal log containing a problem type, a field identifier and original data; and classifying and storing the abnormal logs into a distributed file system according to timestamps, generating a visual abstract report, analyzing problem types, field identifiers and original data, and calling a predefined repair script library for repair. By means of the method, data quality detection is executed in the data input and output process of the DataX, the detection result is output to the text file, and quality detection can be efficiently and accurately conducted on the data.

The invention relates to an intelligent hierarchical access control method and device for railway knowledge and electronic equipment, and the method comprises the steps: registering key certificates for national railway group nodes, railway bureau group nodes and basic-level station nodes, and generating global trusted identity chain certificates; dividing the to-be-accessed railway knowledge assets into four levels, and establishing a relationship between the security level and the user role; loading a railway exclusive strategy expression, and supporting hierarchical inheritance and timeliness logic combination; in response to a user data access request, based on an identity certificate and role attributes, matching a security level of request data with a user role permission threshold, analyzing a railway service dynamic combination condition, judging a current service scene access permission, generating a decryption key or calling an edge gateway to execute outsourcing decryption; and storing the encrypted data to a distributed file system, and recording data hash, a security level and an access operation log. According to the method, the problems of weak identity credibility mechanism, low efficiency of authority management coarse granularity and insufficient industrial characteristic adaptation are solved.

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for securing data. One of the methods includes receiving one or more network data packets. The one or more network data packets include a token that identifies stored sensitive data. The one or more network data packets are desanitized, by: identifying and extracting, from the one or more network data packets, the token; requesting, from a distributed file system, the stored sensitive data, based upon the token; and receiving, in response to the request, the stored sensitive data as received stored sensitive data.

In various examples, data storage is managed using a distributed storage management system that is resilient. Data blocks of a logical block device may be distributed across multiple nodes in a cluster. The logical block device may correspond to a file system volume associated with a file system instance deployed on a selected node within a distributed block layer of a distributed file system. Each data block may have a location in the cluster identified by a block identifier associated with each data block. Each data block may be replicated on at least one other node in the cluster. A metadata object corresponding to a logical block device that maps to the file system volume may be replicated on at least another node in the cluster. Each data block and the metadata object may be hosted on virtualized storage that is protected using redundant array independent disks (RAID).

The invention discloses an automobile production data tracing method, system and device based on a block chain and a medium, and the method comprises the steps: obtaining a product identifier of a production process, original production data and key metadata, storing the original production data to a distributed file system, and generating a unique hash value; packaging the unique hash value and the key metadata to generate a data block, generating block index information according to the unique hash value and the product identifier, and uploading the data block and the block index information to a block chain; obtaining a target hash value and a target data block through indexing on the block chain according to the target product identifier, and determining target metadata according to the target data block; and searching in the distributed file system according to the target hash value to obtain target production data, and generating a target tracing record according to the target production data and the target metadata. The automobile production data tracing efficiency and the accuracy, real-time performance and safety of the automobile production data are improved, and the automobile production data tracing method can be applied to the technical field of automobile production.

The embodiment of the invention discloses an access method in a distributed file system, computing equipment and file access equipment. The method is applied to a first metadata node, the first metadata node belongs to the multiple metadata nodes, the same directory information is stored in each metadata node in the multiple metadata nodes, and the directory information is used for storing directory metadata of the distributed file system. The method comprises the following steps: receiving a first access request from a client, wherein the first access request comprises an object path to be accessed; determining first file fragment information corresponding to first metadata according to the directory information and the to-be-accessed object path; determining an operation result corresponding to the first metadata according to the first file fragment information and the fragment routing information of the first metadata node; and sending the operation result to the client.

The invention relates to the technical field of big data, and discloses a massive small file reading optimization method, system and device and a medium, and the method comprises the following steps: scanning small files in a distributed file system by using a Spark calculation engine to obtain metadata information of the small files; based on the metadata information, small files are classified, and the small files with the same type and data relevance are classified into one group; the resource states of cluster nodes are monitored in real time, the resource states comprise the processor utilization rate, the memory occupancy rate and the I / O load, the load score of each node is calculated, and small file task quotas are dynamically allocated; executing small file merging operation in parallel by utilizing a Spark memory calculation engine to generate a merged file containing a plurality of original small files; creating and maintaining index information in an external database, and recording the position and attribute of each original small file in the combined file; and providing a read-write interface through the special service layer, and positioning and accessing specific small file contents in the combined file.

The invention discloses a distributed file system metadata index optimization method based on remote direct memory access, which comprises directory metadata index based on path name hash: defining a plurality of layers of directories in a directory tree as key directories, and regarding the path as a character string. After calculation of the hash function, determining the storage position of the data in the memory of the metadata server of the distributed file system; and optimizing the memory position of the file metadata based on the popularity, namely placing the file metadata which is frequently accessed at a position adjacent to the key directory metadata to which the file metadata belongs. A metadata management module in a metadata server of the distributed file system regularly checks the popularity of the file and carries out cold and hot migration according to the situation; and client cache optimization oriented to key directory metadata: a metadata cache of a distributed file system client caches the metadata and directory entries of the key directory. Optimally assuming that the metadata and directory entries of the key directory in the cache are not invalid, and updating the cache according to the result of the next-level metadata search. According to the method, the remote reading performance of the metadata index of the distributed file system is optimized by fully utilizing the characteristic of remote direct memory access, the network transmission frequency is reduced, and the load of the metadata server is effectively reduced.

The invention discloses an on-chain and off-chain collaborative interaction system based on a block chain, and the system comprises a data structure module which is used for designing a composite data structure containing metadata, data content, a hash value and a timestamp, and employs a Merkle tree to manage data; the distributed storage and transmission module is used for storing data under the chain by adopting a distributed file system, storing key data on the chain by using a block chain, and designing a secure transmission protocol; by means of the Merkle tree and Hash verification, data integrity can be quickly and accurately verified, tampering can be perceived, a distributed storage and transmission technology and a two-way synchronization strategy can be designed, on-chain and off-chain efficient collaboration is achieved, data processing efficiency and reliability are improved, data security privacy is guaranteed by means of block chain characteristics and encryption, digital signature and other technologies, and the method is suitable for large-scale popularization and application. Distributed storage and a P2P network facilitate node and capability expansion to adapt to data growth, an intelligent contract is introduced, online and offline data collaborative automatic management is achieved, risks are reduced, and efficiency is improved.

The invention provides a Hadoop cluster incremental backup recovery method based on data block optimization, and relates to the technical field of data management, and the method comprises the following steps: obtaining distribution state information and historical change records of data blocks in a distributed file system, extracting space-time correlation characteristics, and constructing a data block dependency graph containing dependency direction and dependency intensity; identifying the changed data blocks, performing topological sorting, and identifying key path data blocks and independent data block clusters; calculating a distribution entropy value according to a copy distribution position, and determining a backup granularity type; a backup execution sequence is generated according to a topological sorting result, key path data blocks are backed up preferentially, independent data block clusters are backed up in parallel, and backup metadata containing dependency relationship identifiers and distribution entropy values are generated; and performing dependency consistency verification after recovery based on the dependency relationship identifier. According to the method, the backup recovery efficiency is improved, the data consistency is guaranteed, and the storage overhead is reduced.

The invention discloses a distributed caching method and system and an acceleration method for large model reasoning service. The system comprises a confidential virtual machine cluster, a zero-trust distributed caching system and a FUSE client. The zero-trust distributed cache system comprises a data strategy service, a data engine plug-in, a distributed file system and a trusted access service. And the FUSE client is deployed at the confidential container end, and establishes secure connection with the trusted access service based on the remote proof and the identity legality confirmation proof of the corresponding confidential container so as to provide a plaintext data access service for the corresponding confidential container. According to the scheme, a distributed cache implementation scheme is provided for the confidential computing container cluster, and the problem of performance loss caused by repeatedly reading, encrypting and decrypting the same data from an external storage system is avoided while the data security is guaranteed by using the large-scale confidential container cluster; and the storage access efficiency in scenes such as model training and reasoning is improved.

The invention discloses an intelligent elevator information management system based on computer vision. The intelligent elevator information management system comprises a data sensing module, a vision detection module, a strategy network construction module and an information management module. The invention relates to the technical field of elevator intelligent management, in particular to an intelligent elevator information management system based on computer vision, which realizes passenger tracking and accurate counting by adopting a time-space spectrum threshold value guided by optical flow to perform optimal cutting and fusing visual detection confidence and Kalman filtering posterior probability; double attention and multi-scale residual error updating are introduced, and short-time passenger flow high-precision prediction is achieved; based on incremental hierarchical clustering, Poisson load estimation and a strategy network, elevator scheduling is dynamically optimized, and waiting time and energy consumption are reduced; and a relational database and distributed file system mixed storage architecture is constructed through a real-time data display single information management aspect, so that the data consistency, the query access efficiency and the decision support capability are enhanced.

The invention discloses a single-file concurrent read-write method and system of a distributed file system. The method comprises the steps that S1, a conservative write range lock application request of a preset interval is sent to a metadata server; the range lock takes a block as the minimum granularity; s2, segmenting to-be-written data into a plurality of data blocks according to a preset block length, and applying for a corresponding locking interval in a local memory lock table according to a starting position and an ending position of each data block in a target file; judging whether the currently applied locking interval is overlapped with the registered locking interval or not; s3, if the data block is not in the write range lock, applying for an interval for expanding the write range lock from the metadata server; s4, distributing the segmented data block write-in requests to block engines of corresponding storage nodes, so that the block engines add data block mutual exclusion locks at corresponding positions; and S5, in response to a request of closing or refreshing the target file by the user, sending a request of releasing the write range lock interval. According to the method and the device, the performance is not reduced on the premise of improving the strong consistency.

The invention discloses a blockchain data security storage method based on verifiable secret sharing. The method comprises the following steps executed by computer hardware: receiving to-be-stored data, performing encryption processing on the to-be-stored data to obtain a ciphertext, storing the ciphertext to a distributed file system, obtaining a storage address, and calculating a hash value of the ciphertext; dividing the encryption key into a plurality of sub-key fragments according to set parameters, and calculating a hash value of each sub-key fragment; packaging the sub-key fragment, the sub-key fragment hash value and the ciphertext hash value into a transaction unit, and writing the transaction unit into the block chain; receiving a decryption request, and reconstructing an encryption key when the number of submitted effective sub-key fragments reaches a threshold value; and acquiring the ciphertext from the distributed file system, verifying the hash value of the ciphertext, and decrypting by using the reconstructed encryption key to obtain the plaintext. According to the invention, the security management and control of the full life cycle of the data are realized, the integrity, confidentiality and authenticability of the data and the secret key are ensured, and the security of the data in the processes of storage, transmission, secret key management and the like is ensured.

The invention discloses a Kubernetes-based multi-tenant resource dynamic isolation method, which comprises the following steps of: constructing a CPU (Central Processing Unit) resource control group by utilizing cgropsv2, and realizing dynamic allocation of CPU resources in combination with a dynamic binding engine; constructing a physically isolated storage pool based on the Ceph distributed file system; the network namespace and eBPF flow control technology is utilized, and LWTunnel packaging is combined to realize tenant network complete isolation and QoS speed limitation; constructing an LSTM neural network prediction model to pre-judge a resource demand; a multi-level security system is constructed through AES-256 encryption, Seccomp / BPF container limitation and other technologies; and defining a dynamic strategy object by utilizing CRD to realize strategy management. Through dynamic adaptive resource scheduling, multi-level security protection and highly flexible customization capability, the cluster resource utilization rate can be improved, the CPU cross-NUMA access delay can be reduced, the malicious traffic interception rate can be improved, and the high-performance and high-security requirements of multi-industry and multi-tenant scenes such as finance and e-commerce can be met.

The invention discloses a trusted data tracing system and method for a prefabricated dish supply chain. The system comprises an edge data acquisition module, an equipment authentication module, an off-chain storage module, an on-chain evidence storage module, a privacy verification module and an intelligent contract execution module. And the edge data acquisition module obtains state data of the prefabricated dishes at the service node and generates an event log. And the equipment authentication module distributes an equipment identity label and performs digital signature on the event log. And the under-chain storage module encrypts the signature log, stores the encrypted signature log into a distributed file system, and generates a content identifier. And the on-chain evidence storage module writes the content identifier, the device identity identifier, the timestamp and the event type code into the alliance block chain. And the privacy verification module responds to the verification request, extracts numerical parameters, generates zero-knowledge proof located in a preset interval, and submits the zero-knowledge proof to the alliance block chain for verification. And the intelligent contract execution module executes business rule logic, and triggers state judgment or abnormal marking based on an on-chain event.

The invention relates to the field of data storage, and provides a crop phenotype data storage and retrieval method and device, equipment and a storage medium, and the method comprises the steps: obtaining multi-modal data comprising crop phenotype data and metadata information; storing the structured data and the metadata information to a distributed database, and storing the unstructured data to a distributed file system; synchronously storing a retrieval field of the structured data, a retrieval field of the small file data and metadata information of the large file data into a distributed search analysis engine; and under the condition that the data matched with the user retrieval condition does not exist in the memory database, determining a retrieval result based on the user retrieval condition and the distributed search analysis engine. According to the method, the multi-modal crop phenotype data are classified and stored according to different data types, so that the storage efficiency and the retrieval rate of the multi-modal crop phenotype data are improved.

The invention relates to a distributed file system recycle bin performance optimization method, which comprises the following steps: in the execution process of an unlink interface of a Linux file system, sequentially completing recycle bin function starting judgment, file deletability verification and recycle bin storage suitability judgment, and obtaining metadata of a target deleted file parent directory from a server; when a file system is created, a global unified root recycle bin directory is synchronously generated, and when a file is deleted, a corresponding recycle bin subdirectory is created in a target MDS based on a parent directory inode number; an atomic operation is executed through a Linux standard rename interface, the file is moved to a corresponding recycle bin subdirectory, and the operation is limited to the interior of a single MDS; when the file is recovered, extracting an inode number of a parent directory, and iteratively querying metadata to restore an original complete path; and during batch recovery, if the original catalogue does not exist or needs to be recovered to a new catalogue, directly and integrally moving the catalogue corresponding to the recycle bin, otherwise, executing recovery operation one by one. The file deletion and recovery performance is remarkably improved, the resource overhead is reduced, and the data consistency is guaranteed.

The embodiment of the invention discloses an information processing method, and the method comprises the steps: generating a target mark for a first directory after an operation request for the first directory of a distributed file system sent by a client is received; wherein the target mark is used for identifying a first operation which can only be executed currently for the first directory and corresponds to the operation request; and if the type of the first operation is the target type, deleting the target mark after the first operation is executed, so that the server executes a second operation for the first directory. The embodiment of the invention further discloses an information processing device and equipment, a computer storage medium and a computer program product.

The embodiment of the invention provides a processing method and device for an NFS service fault in a distributed file system. In the application, when the NFS server side in the distributed file system executes the file lock operation on the file every time, the corresponding file lock information is persistently stored in the extended attribute of the file for recording, so that the file lock of each NFS server side in the distributed file system has sharing performance; different NFS servers can share the file lock by accessing the record persistently stored in the extended attribute of the file, which is helpful for ensuring the consistency of the file lock during service failover. Whether each NFS server side of the whole system has a service fault is monitored from a global perspective through a main monitoring node and a slave monitoring node in the distributed file system, so that file lock recovery between the NFS server side and each client side corresponding to a fault NFS service is coordinated and replaced during service fault switching; therefore, the consistency of file locks between the substitute NFS server and each client is ensured.

A storage management method and a related device are applied to the technical field of storage. In the application, the storage management device obtains the metadata of the first file system from the first file system, provides the metadata of the first file system for the second file system, and enables the data to flow between the first file system and the second file system through the storage interface provided by the first file system; the first file system is a distributed file system. According to the method and the device, metadata flow and data flow between the distributed file system and other file systems can be realized, the problem of data islands of the distributed file system is solved, and the convenience of data access is improved.

A computing system and method for mapping a migration to a new distributed file system is provided. A method may include receiving file system metadata from an existing distributed file system, comparing permissions for each file and each user, generating a plurality of recommended groups, recommended folders, and recommended permissions, and presenting the recommended groups, recommended folders, and recommended permissions to a user.

The invention provides a distributed data storage method, device and equipment and a readable storage medium, and the method comprises the steps: responding to an event created by a storage target, obtaining metadata of the storage target, and submitting the metadata to a key-value pair database in a key-value pair manner; a distributed file system is called to split the specific file into distributed storage objects, and the distributed storage objects obtained through splitting are sent to object storage equipment of a distributed storage cluster; and in response to an event of storage target change, retrieving the corresponding key value pair and modifying the metadata recorded by the corresponding key value pair according to the change condition. Through the technical scheme of the specification, the metadata is stored by the key-value pair granularity, the problem of directory-level read-write amplification is relieved, single file metadata update only needs to modify the corresponding key-value pair, overall directory rewriting is avoided, and IO consumption and network delay are reduced in combination with a write-in merging mechanism of an embedded database and the transparent object splitting capability of a distributed file system.

The invention provides a method and a system for realizing cross-platform business rule dynamic analysis and centralized management based on GraalVM, business rule scripts of different code languages are generated through a rule configuration center and are stored in a MinIO distributed file system in a file form, and a business system front end obtains the business rule script compatible with a current terminal and executes local verification; the back-end service creates an isolated and safe execution environment by integrating a GraalVM multi-language runtime engine, loads a business rule script which is completely the same as that of the front end from a MinIO distributed file system, and executes homologous rule verification; and when the business rules are changed, the rule configuration center updates the business rule scripts and stores the business rule scripts in the MinIO distributed file system in a file form. According to the invention, the problems of logic repetition, maintenance difficulty, update lag and the like caused by repeated coding of service rules at the front end, the rear end and various clients in the existing service system are solved.

The invention discloses a composite file management system and method. According to the system, a three-in-one collaborative architecture is formed by integrating samba service, storage service and SaaS service. Wherein the samba service adopts a five-layer modular design, the network communication module supports an SMB / CIFS protocol to perform data interaction with a multi-platform terminal, the request processing module realizes high-concurrency access control through an asynchronous queue mechanism, and the service logic module is internally provided with file version management, authority verification and log auditing functions. The file storage calling module is in butt joint with a heterogeneous storage system in an adapter mode and supports mixed deployment of a distributed file system and object storage, and the SaaS system calling module is deeply integrated with third-party application service through RESTful API to achieve cross-system file circulation and metadata synchronization. According to the method, transparent scheduling of local storage and cloud storage is achieved through a unified file operation interface, and an enterprise-level file management solution capable of being elastically expanded is constructed.

The invention provides a quota resource processing method which can be applied to the technical field of big data. The method comprises the following steps: loading batch transaction data files to a distributed file system; performing parallel processing on the transaction data through a distributed calculation engine, and calculating a quota resource value corresponding to each transaction; calling an online interface to execute quota resource updating operation, and obtaining an interface calling result in real time; if the quota resource update fails, writing the failure type identifier, the user identifier, the transaction identifier and the quota resource value into a quota resource update failure record table; in response to the update of the quota resource update failure record table, reading the abnormal type identifier; if it is determined that the exception type is system exception according to the exception type identifier, executing quota resource retransmission operation; if it is determined that the exception type is constraint overrun according to the exception type identifier, monitoring a quota resource quota change event related to the target object; and in response to triggering of the quota resource quota change event, executing quota resource reissuing operation.

The invention discloses a file lock management method and device for a distributed file system, equipment and a medium, and is applied to the technical field of computers, and the method comprises the following steps: after receiving a first file access request initiated by a client to a first access node, constructing first file lock information according to an identification parameter in the access request, persistently storing the first file lock information to a shared disk space and caching the first file lock information to a local first access node; the access node is a mounting node for the client to access the distributed file system; switching the mounting node from the first access node to a second access node, and clearing the first file lock information locally cached by the first access node; and after a second file access request initiated by the client to the second access node is received, querying the first file lock information in the shared disk space based on the identification parameter in the access request so as to locally reconstruct the file lock cache at the second access node. According to the method and the device, the problem of file lock loss and residue in a distributed cluster access node switching scene is solved.

A load-aware directory migration method in a distributed file system (DFS), including: 1) receiving a plurality of directory requests created by DFS clients, parsing each of the plurality of directory requests, to obtain directory request information corresponding to the plurality of directory requests; 2) inputting all of the directory request information corresponding to the plurality of directory requests obtained in 1) into a pre-trained machine learning model, to obtain directory labels corresponding to all directory requests; 3) obtaining a label corresponding to a maximum score interval from the directory labels corresponding to all directory requests obtained in 2), and obtaining all directories in the directory request information corresponding to the label; 4) building a directory migration queue based on all the directories obtained in 3); and 5) migrating all the directories in the directory migration queue obtained in 4) through particle swarm optimization (PSO), to obtain a final migration result.

The present disclosure provides methods and apparatuses for writing and searching vector data in a vector database. An example method includes receiving a write request for writing target vector data into a vector database, adding index update information corresponding to the target vector data to a first index of the vector database in a memory table, where the first index is a memory-based real-time read-write vector graph index, in response to flushing the memory table to a disk, writing the index update information of the first index in the memory table into a distributed file system, and converting the first index in the distributed file system into a second index, where the second index is a disk-based vector graph index.