276 results about "Distributed File System" patented technology

The invention provides a Hadoop cluster incremental backup recovery method based on data block optimization, and relates to the technical field of data management, and the method comprises the following steps: obtaining distribution state information and historical change records of data blocks in a distributed file system, extracting space-time correlation characteristics, and constructing a data block dependency graph containing dependency direction and dependency intensity; identifying the changed data blocks, performing topological sorting, and identifying key path data blocks and independent data block clusters; calculating a distribution entropy value according to a copy distribution position, and determining a backup granularity type; a backup execution sequence is generated according to a topological sorting result, key path data blocks are backed up preferentially, independent data block clusters are backed up in parallel, and backup metadata containing dependency relationship identifiers and distribution entropy values are generated; and performing dependency consistency verification after recovery based on the dependency relationship identifier. According to the method, the backup recovery efficiency is improved, the data consistency is guaranteed, and the storage overhead is reduced.

The invention discloses a single-file concurrent read-write method and system of a distributed file system. The method comprises the steps that S1, a conservative write range lock application request of a preset interval is sent to a metadata server; the range lock takes a block as the minimum granularity; s2, segmenting to-be-written data into a plurality of data blocks according to a preset block length, and applying for a corresponding locking interval in a local memory lock table according to a starting position and an ending position of each data block in a target file; judging whether the currently applied locking interval is overlapped with the registered locking interval or not; s3, if the data block is not in the write range lock, applying for an interval for expanding the write range lock from the metadata server; s4, distributing the segmented data block write-in requests to block engines of corresponding storage nodes, so that the block engines add data block mutual exclusion locks at corresponding positions; and S5, in response to a request of closing or refreshing the target file by the user, sending a request of releasing the write range lock interval. According to the method and the device, the performance is not reduced on the premise of improving the strong consistency.

The invention discloses a trusted data tracing system and method for a prefabricated dish supply chain. The system comprises an edge data acquisition module, an equipment authentication module, an off-chain storage module, an on-chain evidence storage module, a privacy verification module and an intelligent contract execution module. And the edge data acquisition module obtains state data of the prefabricated dishes at the service node and generates an event log. And the equipment authentication module distributes an equipment identity label and performs digital signature on the event log. And the under-chain storage module encrypts the signature log, stores the encrypted signature log into a distributed file system, and generates a content identifier. And the on-chain evidence storage module writes the content identifier, the device identity identifier, the timestamp and the event type code into the alliance block chain. And the privacy verification module responds to the verification request, extracts numerical parameters, generates zero-knowledge proof located in a preset interval, and submits the zero-knowledge proof to the alliance block chain for verification. And the intelligent contract execution module executes business rule logic, and triggers state judgment or abnormal marking based on an on-chain event.

The invention relates to a distributed file system recycle bin performance optimization method, which comprises the following steps: in the execution process of an unlink interface of a Linux file system, sequentially completing recycle bin function starting judgment, file deletability verification and recycle bin storage suitability judgment, and obtaining metadata of a target deleted file parent directory from a server; when a file system is created, a global unified root recycle bin directory is synchronously generated, and when a file is deleted, a corresponding recycle bin subdirectory is created in a target MDS based on a parent directory inode number; an atomic operation is executed through a Linux standard rename interface, the file is moved to a corresponding recycle bin subdirectory, and the operation is limited to the interior of a single MDS; when the file is recovered, extracting an inode number of a parent directory, and iteratively querying metadata to restore an original complete path; and during batch recovery, if the original catalogue does not exist or needs to be recovered to a new catalogue, directly and integrally moving the catalogue corresponding to the recycle bin, otherwise, executing recovery operation one by one. The file deletion and recovery performance is remarkably improved, the resource overhead is reduced, and the data consistency is guaranteed.

The invention provides a method and a system for realizing cross-platform business rule dynamic analysis and centralized management based on GraalVM, business rule scripts of different code languages are generated through a rule configuration center and are stored in a MinIO distributed file system in a file form, and a business system front end obtains the business rule script compatible with a current terminal and executes local verification; the back-end service creates an isolated and safe execution environment by integrating a GraalVM multi-language runtime engine, loads a business rule script which is completely the same as that of the front end from a MinIO distributed file system, and executes homologous rule verification; and when the business rules are changed, the rule configuration center updates the business rule scripts and stores the business rule scripts in the MinIO distributed file system in a file form. According to the invention, the problems of logic repetition, maintenance difficulty, update lag and the like caused by repeated coding of service rules at the front end, the rear end and various clients in the existing service system are solved.

The invention provides a quota resource processing method which can be applied to the technical field of big data. The method comprises the following steps: loading batch transaction data files to a distributed file system; performing parallel processing on the transaction data through a distributed calculation engine, and calculating a quota resource value corresponding to each transaction; calling an online interface to execute quota resource updating operation, and obtaining an interface calling result in real time; if the quota resource update fails, writing the failure type identifier, the user identifier, the transaction identifier and the quota resource value into a quota resource update failure record table; in response to the update of the quota resource update failure record table, reading the abnormal type identifier; if it is determined that the exception type is system exception according to the exception type identifier, executing quota resource retransmission operation; if it is determined that the exception type is constraint overrun according to the exception type identifier, monitoring a quota resource quota change event related to the target object; and in response to triggering of the quota resource quota change event, executing quota resource reissuing operation.

The invention discloses a file lock management method and device for a distributed file system, equipment and a medium, and is applied to the technical field of computers, and the method comprises the following steps: after receiving a first file access request initiated by a client to a first access node, constructing first file lock information according to an identification parameter in the access request, persistently storing the first file lock information to a shared disk space and caching the first file lock information to a local first access node; the access node is a mounting node for the client to access the distributed file system; switching the mounting node from the first access node to a second access node, and clearing the first file lock information locally cached by the first access node; and after a second file access request initiated by the client to the second access node is received, querying the first file lock information in the shared disk space based on the identification parameter in the access request so as to locally reconstruct the file lock cache at the second access node. According to the method and the device, the problem of file lock loss and residue in a distributed cluster access node switching scene is solved.

A load-aware directory migration method in a distributed file system (DFS), including: 1) receiving a plurality of directory requests created by DFS clients, parsing each of the plurality of directory requests, to obtain directory request information corresponding to the plurality of directory requests; 2) inputting all of the directory request information corresponding to the plurality of directory requests obtained in 1) into a pre-trained machine learning model, to obtain directory labels corresponding to all directory requests; 3) obtaining a label corresponding to a maximum score interval from the directory labels corresponding to all directory requests obtained in 2), and obtaining all directories in the directory request information corresponding to the label; 4) building a directory migration queue based on all the directories obtained in 3); and 5) migrating all the directories in the directory migration queue obtained in 4) through particle swarm optimization (PSO), to obtain a final migration result.

The present disclosure provides methods and apparatuses for writing and searching vector data in a vector database. An example method includes receiving a write request for writing target vector data into a vector database, adding index update information corresponding to the target vector data to a first index of the vector database in a memory table, where the first index is a memory-based real-time read-write vector graph index, in response to flushing the memory table to a disk, writing the index update information of the first index in the memory table into a distributed file system, and converting the first index in the distributed file system into a second index, where the second index is a disk-based vector graph index.

The invention discloses a distributed file system metadata full-memory management method and system.The distributed file system metadata full-memory management method comprises the steps that distributed file system metadata are stored in a memory and a storage layer through the same layout, and the same layout means that memory pages in the memory and storage blocks in the storage layer are the same in size; and taking the memory page in the memory as a page cache of a corresponding storage block in the storage layer, responding to a read request and a write request of a client through the distributed file system metadata stored in the memory, and directly writing the memory page written by the write request into the storage layer when the write request is received so as to realize persistent storage of the distributed file system metadata. The distributed file system metadata full-memory management method aims at achieving distributed file system metadata full-memory management, read operation completely gets rid of the I / O bottleneck, all storage bandwidths are dedicated to write operation, memory bandwidths are fully utilized for distributed file system metadata management, and persistence overhead is reduced.

The invention discloses a CAE software management system and method, and relates to the technical field of CAE software management, and the method comprises the steps that a graphic rendering virtualization module provides a virtual desktop environment, so that a user interacts with a CAE interface based on the virtual desktop environment to generate a task description file; the job management and monitoring module receives the task description file and performs standardization processing on the task description file to obtain a standard task description file; the computing resource intelligent scheduling module performs resource matching on the standard task description file through a scheduling algorithm to determine a target computing node, and processes the CAE simulation task through the cross-platform scheduler adapter module and the standard task description file according to the target computing node; and when the computing resource intelligent scheduling module processes the CAE simulation tasks, the data management and security control module performs data synchronization through a distributed file system, creates an independent virtual sandbox environment for each CAE simulation task, and binds a sandbox with a target computing node.

The invention relates to a virtual consistency multi-node message passing interface expansion method, device and equipment. The method comprises the following steps: deploying a distributed file system client at each computing node of a multi-node cluster, creating a unified virtual directory mounting point, transparently converting a file operation request under the unified virtual directory mounting point into a corresponding network protocol, and docking a global unified data storage pool, the unified virtual directory mounting point is used for providing consistent file views for all the computing nodes; pointing a file read-write path prefix of the message passing interface process to the unified virtual directory mounting point; and in response to the received user operation request, allocating computing nodes to the user operation request, ensuring that all the allocated computing nodes are mounted to the distributed file system, and starting a message passing interface process. By adopting the method, the consistency of the cross-node file view can be solved from the system level on the premise of not changing the communication logic.

The invention discloses a multi-source heterogeneous live-action three-dimensional database construction method and device and a medium, and relates to the field of database construction. The method comprises the steps of obtaining multi-source heterogeneous data; performing standardization processing on the multi-source heterogeneous data to obtain standardized multi-source heterogeneous data; the method comprises the following steps: constructing an entity classification set, an entity attribute set and an entity relationship set based on a real scene three-dimensional data classification system, and constructing an RDF triple by using a graph convolutional neural network; and constructing a multi-source heterogeneous live-action three-dimensional database based on the standardized multi-source heterogeneous data and the RDF triple. According to the method, the complexity of data conversion is reduced through the preset data classification system and the standardized naming rule, and the automation level of data matching and storage is improved in combination with the knowledge graph driven by priori cognition. Meanwhile, a mixed framework of a relational database, a distributed file system and a graph database is adopted, a multi-stage index structure is combined, efficient storage and rapid retrieval of multi-modal data are supported, and the query time is remarkably shortened.

The invention discloses an intelligent data verification method and device and related equipment. The method comprises the steps of obtaining a field type, a partition key and a partition data size of a target Hive table; generating a field filtering expression based on the rule engine and the field type of the target Hive table; generating a cascade check condition based on the partition key of the target Hive table; generating a verification rule based on the field filtering expression and the cascade verification condition; fragmenting the data of each partition based on a storage space of a storage block on the distributed file system and the size of the data of each partition; and verifying each piece of fragmented data to obtain a target global check code. According to the method and the device, when the Hive data before and after migration are verified, the non-scalar fields in the Hive table are filtered out by setting the field filtering expression, so that the tolerance to small errors during verification is improved; when the verification is executed, each piece of partition data is fragmented, and each piece of fragmented data is verified in parallel based on the plurality of computing nodes of the distributed cluster, so that the verification efficiency can be improved.

A distributed file system unit (200) receives, from a user, a request for a file divided into a plurality of fragments and stored unencrypted in a distributed file system, requests the plurality of fragments of the requested file requested by the user from one or more terminals connected to the distributed file system, receives the plurality of fragments, which are encrypted using a public key belonging to the owner of the requested file, from the one or more terminals, decrypts the plurality of fragments from the encrypted plurality of fragments using a secret key belonging to the user, and forms a file using the plurality of decrypted fragments.

The invention discloses an HDFS (Hadoop Distributed File System) intelligent hierarchical storage method and system based on multi-dimensional popularity perception and tape library integration, and belongs to the technical field of distributed file systems, and the method comprises the following steps: expanding HDFS metadata to construct a unified namespace; acquiring data block access features and dynamically dividing hot, warm and cold data by using a K-means clustering algorithm; executing a self-adaptive grading strategy: adding a copy to hot data and storing in an SSD (Solid State Disk), maintaining a default copy to warm data and storing in a disk, and migrating cold data to a tape library and storing by adopting an erasure code; scattered storage and fault tolerance of relevance perception are realized through a message queue; metadata persistence and periodical re-evaluation are carried out, and automatic migration and dynamic adjustment of data among the SSD layer, the disk layer and the tape library layer are achieved. By adopting the method and the system, the storage strategy and the data access value are accurately matched, the storage cost is finally reduced, and the system I / O performance and the resource utilization rate are improved.

The embodiment of the invention provides a big data analysis system, and relates to the technical field of big data, and the system comprises a container meta-management platform which is used for constructing and managing a plurality of container cloud clusters, and creating a tenant corresponding to each user so as to configure each tenant to the required container cloud cluster; the plurality of big data clusters are used for calculating the submitted corresponding data query tasks; according to the multi-tenant platform, a service layer is used for transmitting a data query task to a calculation engine layer under the condition that the data query task submitted by a user is acquired; the calculation engine layer is used for submitting the data query task to a corresponding big data cluster for calculation; the distributed file system is used for storing the read-write permission of each tenant so as to establish connection with the corresponding object storage cluster according to the read-write permission of the tenant corresponding to the data query task; and the object storage system comprises a plurality of object storage clusters for storing read-write data required by the corresponding tenants.

The invention discloses a distributed lock management method and device, electronic equipment and a storage medium, and relates to the technical field of storage, the management method comprises the following steps: in response to lease overdue of a client, judging whether a server works normally or not; in response to normal work of the server, releasing a distributed lock held by the overdue client, and recording session information of the overdue client into a preset database to prohibit the overdue client from executing lock recovery; and in response to restarting of the server, notifying the metadata server to enter a delay period, determining a target client of which the lock state is not recovered after the delay period is ended, controlling the metadata server to clear lock information of a distributed lock held by the target client, and recording session information of the target client into a preset database, the technical problem that the unconditionally trusted client recovers the lock request after the server is restarted is solved, and the technical effect of guaranteeing the data consistency and reliability of the distributed file system in a fault scene is achieved.

The invention discloses a power grid equipment fault analysis method and device, computer equipment and a storage medium, and relates to the technical field of power grid equipment fault diagnos.The power grid equipment fault analysis method and device integrate various complex and diverse data sources which are widely distributed in a power grid, construct a highly structured and logically clear data architecture based on a snowflake model, and analyze the fault of the power grid equipment. And data related to fault analysis can be quickly positioned from mass data in a short time. Meanwhile, data storage is carried out in the Hadoop distributed file system, the information dimension of fault analysis is greatly enriched by utilizing the powerful distributed storage, data redundancy and fault-tolerant mechanism of the Hadoop distributed file system, and a stable data basis is provided for power grid equipment fault analysis. And multiple complex factors of power grid equipment faults are comprehensively considered, a comprehensive and high-adaptability model system is constructed, various complex fault types can be accurately identified, fault diagnosis errors caused by single or inaccurate models are effectively avoided, and the fault prediction and diagnosis capability is improved especially under the complex conditions of severe weather and the like.

The application belongs to the field of data storage, and discloses a directory distribution method, an electronic device and a computer readable storage medium. The method comprises the following steps: inserting a to-be-created directory item into a virtual directory of a virtual set of a distributed file system based on a skip consistent hashing algorithm; if the storage amount of the virtual directory is greater than a preset storage threshold, determining whether the number of sets of the virtual set in the distributed file system is less than a preset set threshold; if yes, splitting a new directory set from the virtual set and allocating a metadata node to the new directory set; if no, determining whether the number of directories of the virtual directory is less than a preset directory threshold; if the number of directories is less than the preset directory threshold, performing directory splitting based on a global bitmap of the virtual set where the virtual directory is located; and if the number of directories is greater than or equal to the preset directory threshold, controlling the directory expansion and the directory splitting to be alternately split. The application aims to solve the technical problem that the creation of a directory in a distributed file system is not efficient enough.

The application belongs to the technical field of blockchains, and particularly relates to a compliance data credible storage and traceability system based on digital fingerprints and alliance chains, which comprises a data preprocessing and structured slicing module, a multi-level digital fingerprint construction module, an alliance chain consensus storage module and a compliance verification and traceability module. The data preprocessing and structured slicing module is used for data standardization and slicing. The multi-level digital fingerprint construction module is used for constructing a Merkle tree based on a slicing fingerprint and extracting a global fingerprint. The alliance chain consensus storage module is used for encapsulating the global fingerprint and metadata and storing encrypted slices in a distributed file system. The compliance verification and traceability module is used for realizing data verification and tampering accurate positioning by using a Merkle path proof. The multi-level hash structure improves large-scale data verification efficiency, the selective disclosure mechanism solves the conflict between privacy protection and compliance audit, and the alliance chain consensus mechanism ensures the whole life cycle safety and credible traceability of electronic data storage.

A load-aware directory migration method in a distributed file system (DFS), including: 1) receiving a plurality of directory requests created by DFS clients, parsing each of the plurality of directory requests, to obtain directory request information corresponding to the plurality of directory requests; 2) inputting all of the directory request information corresponding to the plurality of directory requests obtained in 1) into a pre-trained machine learning model, to obtain directory labels corresponding to all directory requests; 3) obtaining a label corresponding to a maximum score interval from the directory labels corresponding to all directory requests obtained in 2), and obtaining all directories in the directory request information corresponding to the label; 4) building a directory migration queue based on all the directories obtained in 3); and 5) migrating all the directories in the directory migration queue obtained in 4) through particle swarm optimization (PSO), to obtain a final migration result.

The application discloses a multi-domain metadata management method for a high-performance computing scene, and is applied to a distributed file system, and the method comprises the following steps: a multi-domain metadata register acquires and saves a workflow name, a storage node list file and a file system priority configuration file of each temporary file system; the multi-domain metadata register waits for and acquires a view fusion request of the temporary file system needing to be fused, and reads the corresponding storage node list file and the file system priority configuration file according to the workflow name of the temporary file system needing to be fused to perform view fusion, so as to provide a unified metadata view. The application dynamically constructs a metadata view by using a differentiated method, decouples consistency constraints among each other, flexibly adjusts a management strategy of metadata according to different requirements, reduces global synchronization management overhead and index overhead of metadata, and further effectively improves concurrent access performance of metadata, and consumption of network and storage resources is reduced.

The invention discloses a distributed file system oriented to consumer-level equipment and a data security management method of the distributed file system, and belongs to the technical field of computer systems. The system comprises a virtual namespace management module used for maintaining a virtual namespace tree formed by combining a local main namespace tree and a remote snapshot sub-tree; the access control and encryption module is used for executing access control based on the attribute set of the virtual namespace tree nodes; and the data processing pipeline module is used for partitioning the file and realizing pipeline operation of transmission and decryption through independent data flow and control flow channels. According to the invention, the access control and the encryption strategy are tightly coupled based on the unified attribute set, so that file re-encryption is not needed when the authority is changed, and the system flexibility is improved; and the security operation delay is hidden through data partitioning and pipeline processing, so that the file access efficiency is improved, and the problem of balance among security, flexibility and high efficiency of the distributed file system of the consumption equipment is effectively solved.

The invention discloses an abnormity monitoring method, system, equipment and medium based on a plant station gateway metering point, and relates to the technical field of gateway metering point abnormity detection, and the method comprises the steps: collecting the real-time data of the plant station gateway metering point through an industrial protocol interface, loading a protocol stack in a collection gateway, carrying out the preprocessing of the preliminarily collected data, and carrying out the detection of the abnormity of the gateway metering point; data transmission is carried out on the preprocessed data through a real-time stream processing layer by utilizing a transmission protocol, the real-time data is stored in a memory database, historical data is stored in a distributed file system, data organization and management are carried out, and multi-mode anomaly detection is carried out on the stored data through an anomaly detection service deployed in a containerization mode. And identifying various abnormities, packaging and outputting a detection result, and dynamically rendering through a responsive front-end interface to display the detection result in real time. According to the invention, the multi-mode intelligent detection and containerized elastic architecture are fused, so that the leap-type improvement of the metering abnormity of the plant station from passive alarm to active early warning and deep diagnosis is realized.

A cross-platform data migration method and a cross-platform data migration system include determining that a first database is compatible with a second database, receiving a user request associated with an instruction to migrate data from the first database to the second database, determining that backup data corresponding to a current state of the first database is available, mounting the backup data onto the second platform via a distributed file system protocol, converting the backup data from the source endian format to a target endian format, and restoring the backup data in the target endian format to the second platform.

The invention discloses a control method and device of a distributed file system and a storage medium, and relates to the technical field of data storage, the control method of the distributed file system comprises the steps of determining the sequence of identifiers of all hard disks based on a first random number, and filling all the identifiers into a candidate hard disk list according to the sequence; calculating a hard disk weight value of each identifier according to the space utilization rate and the file quantity corresponding to each identifier; and rearranging the identifiers in the candidate hard disk list according to the hard disk weight value to obtain a hard disk list. The technical effects of balancing space balance and performance and improving the overall performance of the distributed file system are achieved.

This application discloses a locking method, apparatus, device, and medium for a file system, relating to the field of distributed file systems. The method includes: receiving a target I / O request; determining the type of the target requested resource in the current file system corresponding to the target I / O request; if the target requested resource is a file structure type, adding a structure lock to the structure corresponding to the target requested resource; if the target requested resource is an internal module resource type, adding a module lock to the module corresponding to the target requested resource; if the target requested resource is a globally shared resource type, adding a client lock to the data structure corresponding to the target requested resource; wherein the structure lock, module lock, and client lock are all mutex locks. This invention can differentiate the constraint force of client locks using a modular unlocking method, improving file read / write concurrency performance, reducing blocking phenomena caused by client locks, reducing read / write time, improving file system read / write performance, and indirectly improving file read / write bandwidth performance.

The invention relates to the technical field of distributed storage and big data, and discloses a distributed storage and management method and system for clinical multi-source and multi-modal data, and the method comprises the steps: receiving and preprocessing multi-modal clinical data of a plurality of heterogeneous data sources, and obtaining a data package containing a binary file, a source identifier and preliminary metadata; integrating the data type, the label and the preliminary metadata to generate complete metadata; the file is stored in a FastDFS distributed file system, and an MD5 value is calculated; writing the metadata into the MySQL metadatabase, and triggering file deletion when the metadata fails; and performing retrieval in the MySQL metadatabase according to the query condition, obtaining a target file identifier meeting the query condition, generating a file access address based on the target file identifier, and returning the file access address to the user side. According to the invention, the storage efficiency, the sharing cooperation capability and the system performance stability of clinical multi-source and multi-modal data can be improved.

This invention relates to the field of storage system operation and maintenance and optimization technology, and in particular to a method, system, and apparatus for write operation traffic statistics based on kernel mounting. The method includes establishing a mapping relationship between all directories in a distributed file system and clients; identifying at least one target directory accessed by clients within a preset time period; monitoring each target directory, obtaining the storage capacity change value of the target directory within the preset time period, and the number of file handle accesses to the target directory; calculating the write operation traffic of each client to the target directory based on the storage capacity change value and the number of file handle accesses; and implementing at least one optimization strategy based on the write operation traffic of each client to the target directory to improve the stability of the distributed file system. This invention can improve the stability of a distributed file system while using kernel mounting to access it.