598 results about "Distributed memory systems" patented technology

The invention discloses an electronic archive intelligent retrieval method and system based on a block chain, and relates to the technical field of archive retrieval, and the method comprises the steps: extracting text features of an electronic archive, and generating metadata; generating a content hash value from the electronic file original text; writing the metadata, the content hash value and the ABE access strategy into the block chain smart contract; constructing a reverse index based on the metadata semantic tag, wherein an index entry is associated with a block chain storage address; calculating a root hash and anchoring the root hash to the block chain; analyzing a keyword and a digital identity certificate in the user retrieval request; calling an intelligent contract to verify whether the user attribute accords with the ABE access strategy of the target file; retrieving an encrypted hash list matched with the file in the distributed index; acquiring the encrypted file fragments from the distributed storage system; verifying data integrity; and combining the fragments to generate a final retrieval result. The method has the advantages that through deep coupling of the block chain and attribute encryption, an electronic archive management system considering security and intelligent retrieval is constructed.

The invention relates to the technical field of distributed storage, in particular to a high-speed distributed storage system and method, and the system comprises a file index characterization module which carries out the statistics of the access frequency and the recent access timestamp of each data block in the system, and obtains the popularity magnitude of the file data blocks; according to the method, the popularity magnitude concept is introduced on the basis of linkage statistics of the data block access frequency and the recent access time, the data blocks with remarkable service influence in the current system can be recognized, the priority processing sequence of different data blocks in the repair process is divided, and the decision-making efficiency during fault recovery is improved. After the to-be-repaired data blocks are identified, the comprehensive weight values are formed through correlation calculation with the popularity magnitude, and all the weight values are arranged in a descending order, so that dynamic grading and sequential scheduling of repair tasks are realized, and important data blocks are guaranteed to preferentially obtain recovery resource support.

The invention discloses a dynamic migration method and device of data, electronic equipment, a storage medium and a program, and relates to the technical field of distributed storage, and the method comprises the following steps: scanning N pieces of data stored in a distributed storage system through a dynamic hierarchical manager to obtain N pieces of access information; calculating a heat value corresponding to each piece of data according to the access information and a preset heat evaluation function to obtain N heat values, and dividing the N heat values according to set M heat intervals to obtain M data sets; wherein M is smaller than or equal to N, and M and N are positive integers; and carrying out dynamic migration of data storage positions on the N pieces of data based on the M data sets.

The invention discloses a stable transmission method for server data encryption and rapid authentication, and belongs to the technical field of data security. The method comprises the following steps: dividing data into three levels of core sensitive data, important data and common data by utilizing a semantic analysis model; performing encryption protection on different levels of data by adopting a hierarchical encryption strategy; a master key is generated based on a chaotic system, a session key is generated in combination with a timestamp and a device fingerprint and is regularly alternated, and secure distribution is performed through a trusted execution environment and a block chain; multi-modal features are fused, the identity is verified through zero-knowledge proof, and the authentication strength is adjusted through dynamic risk assessment. The method is suitable for cloud computing, the Internet of Things and a distributed storage system, through deep fusion of a multi-level hybrid encryption algorithm, a dynamic key management mechanism and a self-adaptive authentication strategy, data security encryption, rapid authentication and stable transmission are achieved, and security, efficiency and adaptability are improved.

The invention discloses a thematic charting processing method and system based on online data, and the method comprises the steps: carrying out the standardization processing of original data through a pre-established data processing flow, segmenting the original data into tile data, storing the tile data in a distributed storage system, and constructing a structured tile data set. And according to the structured tile data set, performing rendering processing by applying a color matching scheme, and generating a map service interface to provide a dynamic map data stream. The data is loaded to a browser end drawing environment through the map service interface, an initial drawing project file containing a drawing layout is generated, and the drawing layout comprises at least one map frame. And adding map element data into the initial cartographic engineering file and adjusting the position so as to update the cartographic engineering file. Therefore, according to the thematic charting processing method based on the online data, the data processing efficiency is improved, the cooperation threshold is reduced, and dynamic sharing is achieved.

The invention discloses a metadata prefetching method and device, a storage medium, electronic equipment and a program, and relates to the field of metadata management of a distributed storage system.The metadata prefetching method comprises the steps that a first information block corresponding to a metadata operation event in a target storage area of each node is obtained; updating the three-dimensional correlation degree between any two metadata in each node according to the first information group; determining a prefetching priority corresponding to any two pieces of metadata through the updated three-dimensional correlation degree; and generating a heat association list of the metadata in each node according to the prefetching priority, and respectively sending the heat association lists corresponding to the plurality of nodes to the client so as to indicate the client to prefetch the target metadata in the plurality of nodes according to the plurality of heat association lists. Through the method and the device, the problem of low pre-fetching accuracy caused by dependence on static path association of the metadata and poor cross-node adaptability of a pre-fetching method for the metadata in a distributed storage system is solved, and the pre-fetching accuracy is improved.

Various embodiments of the present technology generally relate to systems and methods for providing managing object lock settings during cross-grid replication within distributed storage systems. In an example, ingestion of an object into a first grid of a distributed storage system may be detected. Responsive to detecting ingestion of the object, object lock settings for the object may be determined. Once the object lock settings are determined, the object lock settings may be validated against destination object lock settings. If the destination object lock settings are validated, cross-grid replication of the object may be initiated. During cross-grid replication, the object lock header may be provided in a replication payload transmitted from the first grid to a second grid. When the object is replicated, the destination object lock settings may be determined for the object, which may include the object lock settings as identified in the object lock header.

The invention discloses a resource management method and device, equipment and a medium, and relates to the technical field of storage, the resource management method comprises the following steps: carrying out performance test and data analysis on different storage architectures according to an I / O path of a distributed storage system and an asynchronous programming model, and determining processor static parameters under the different storage architectures according to the results of the performance test and the data analysis; the method comprises the following steps: collecting performance indexes of a distributed storage system through an asynchronous programming model, predicting load change conditions of the system according to the performance indexes of the distributed storage system, and dynamically adjusting processor resources according to the load change conditions of the system; managing a large-page memory pool and a cache according to the large-page memory management model and the business model type of the distributed storage system; and optimizing a memory application and release mechanism in the storage service of the distributed storage system. Through dynamic scheduling of processor resources and a large-page memory pool data sharing mechanism, balanced allocation of storage and calculation resources is realized, and the system performance and the resource utilization rate are improved.

A method and system for updating a topology on router nodes in a distributed storage system are described. The method can include obtaining, by a first service of a control plane service, topology information from an inventory data store by querying a second service of the control plane service that manages the inventory data store for the topology information. The method can further include generating, by the first service of the control plane service, a topology payload based on the obtained topology information. The method can also include sending, by the first service of the control plane service, a request comprising the topology payload to a router node.

The invention provides a cross-domain authentication and data sharing method and system capable of verifying homomorphic encryption, and relates to the technical field of blockchains, comprising the following steps: a data owner generates a public and private key pair by using a homomorphic encryption scheme based on a lattice password; the data owner performs homomorphic encryption on the original data by using the public key to generate a homomorphic ciphertext; the data owner performs integrity protection on the homomorphic ciphertext by using a Merkle tree structure to generate a ciphertext abstract; the cross-domain authentication center receives a data access request sent by the data user, and the cross-domain authentication center verifies the identity of the data user by using a multi-factor authentication method; the data user submits an authorization token and a data access request to the distributed storage system; the distributed storage system verifies the authorization token and returns corresponding homomorphic ciphertext data; the verification network verifies the calculation process, generates a verification report and sends the verification report to the data owner; and the data user collects partial decryption results and reconstructs a complete decryption result.

The invention relates to the technical field of power system dispatching, in particular to a distributed storage system for source-network-load-storage collaborative dispatching, which comprises a distributed data acquisition module, a regional collaborative sensing module, a distributed energy storage evaluation module, a dispatching instruction generation module and a dispatching instruction distribution module, wherein the distributed data acquisition module is used for acquiring real-time operation data of source, network, load and storage nodes in parallel; the regional collaborative sensing module is used for calculating a real-time power deviation value; the distributed energy storage evaluation module is used for generating an energy storage adjustment instruction set; and the scheduling instruction generation module is used for generating a final refined scheduling instruction. According to the invention, through constructing a cooperative scheduling mechanism of the source network load storage full chain, accurate identification of power deviation and optimal distribution of energy storage resources are realized, and scheduling response efficiency and operation stability of the system are significantly improved.

The invention discloses a distributed storage data active migration method and system based on intelligent health prediction, and relates to the technical field of distributed storage system operation and maintaining.The method comprises the steps that multi-dimensional health state indexes of all nodes in a distributed storage cluster are periodically collected; calculating the health score of each node through a multi-time scale dynamic model based on the health state indexes; when the health score of any node is continuously lower than a preset active migration threshold value, marking the node as a high-risk node, and triggering an active migration process for all data copies on the node; constructing a multi-target optimization cost function for the to-be-migrated data, and selecting a candidate node with the lowest comprehensive cost from the healthy node pool as a migration target according to the cost function; and migrating data on the high-risk node to a selected target node at a resource-aware dynamic rate, atomically updating metadata after data migration succeeds, and then asynchronously cleaning old data copies on the high-risk node.

The invention relates to an agent content storage method and device, electronic equipment and a medium, and is applied to a distributed storage system.The method comprises the steps that in response to a session request input by a user, dialogue content corresponding to the session request is obtained, and the dialogue content is stored to first-level equipment, the first-level equipment is used for storing the real-time content generated by the intelligent agent; when it is determined that the session is ended, the session content in the first-level device is migrated to a second-level device, and the second-level device is used for storing the short-term memory content and the group sharing data of the intelligent agent; updating the content of a third-level device based on the access information of the second-level device, wherein the third-level device is used for storing long-term memory and differential memory of the intelligent agent; and the specified content in the third-level device is synchronized to a fourth-level device, and the fourth-level device is used for storing the universal knowledge base and the group unified memory so as to efficiently realize the storage of the agent data.

The invention discloses a failover method and device, electronic equipment and a storage medium, and relates to the technical field of distributed storage, and the failover method comprises the following steps: determining a first disk resource group corresponding to a first storage node and a second disk resource group corresponding to a second storage node; and establishing a double-control relationship between the first disk resource group and the second disk resource group. And after the establishment of the double-control relationship is completed, carrying out fault switching on different disk resource groups under the double-control relationship based on the first mechanism and the second mechanism. The technical problem that in a distributed storage system, efficient and safe disk resource management and data access control of a fault controller cannot be effectively achieved under a double-controller architecture is solved, and the technical effects of improving rapidness, smoothness and automatic back-switching of fault switching and remarkably improving the high availability of the distributed storage system are achieved.

The invention relates to a meter box multi-parameter prediction method and system integrated with big data processing, and the method comprises the steps: collecting the multi-dimensional original data of the operation of a meter box in real time through a built-in RTU collection module of the meter box, encrypting the original data, uploading the encrypted original data to a cloud server, and carrying out the cleaning, standardization and feature extraction, thereby obtaining structured data; storing the structured data to a distributed storage system, and indexing and compressing the data; calling the compressed structured data to construct a multi-task prediction model, and performing supervised training on the multi-task prediction model by using historical statistical data to obtain a trained prediction model; and inputting the original data acquired in real time into the trained prediction model, outputting a single-parameter prediction result, and performing multi-parameter collaborative analysis based on the single-parameter prediction result and the original data to generate a collaborative prediction result.

The invention relates to the field of distributed storage, in particular to a method for safely transmitting and storing file fragments in a distributed storage system, and solves the problem that data are tampered or leaked in the transmission, storage and decryption process in the existing distributed storage field. The method comprises the following steps: step 1, data fragmentation; 2, data are uploaded before being uploaded; step 3, storing; 4, encrypting file link generation and forwarding; and 5, receiving and decrypting the file. According to the method, fragmented data, secret keys and parameters of each storage node in the distributed storage system are mastered by multiple parties, and encryption and decryption processing of organization, transmission and storage of the fragmented data is realized by adopting a stream encryption and parallel processing mode; the transmission and storage security of the file in the distributed system is effectively guaranteed, and the file is prevented from being tampered.

The invention discloses a resource quota adjustment method, device and equipment and a readable storage medium, and is applied to the technical field of distributed storage, and the method comprises the steps: determining current traffic data corresponding to a current service request; performing linear prediction based on the current traffic data to obtain initial predicted traffic, and performing nonlinear residual correction on the predicted traffic to obtain target predicted traffic; and configuring the resource quota based on the target predicted traffic to obtain a target pre-application resource quota. According to the invention, prediction is carried out by using the prediction model based on the current traffic data to obtain the initial predicted traffic, nonlinear residual correction is carried out on the predicted traffic to obtain the target predicted traffic, and the target predicted traffic is obtained through prediction and error compensation, so that the predicted traffic is more accurate, and the prediction efficiency is improved. Therefore, the target pre-applied resource quota obtained by performing pre-application on the resource quota based on the target predicted traffic subsequently is more accurate, the burst traffic resistance of the distributed storage system is improved, and the subsequent service quality is improved.

The invention discloses a data consistency processing method, a distributed storage system and electronic equipment, and belongs to the field of distributed data storage systems, and the data consistency processing method comprises the following steps: when any storage node in the distributed storage system receives to-be-updated data sent by a client, obtaining a corresponding data updating path by taking the storage node as a first storage node; according to the data updating path, each storage node carries out data persistence on itself after receiving the data to be updated, if persistence succeeds, the data to be updated is transmitted to the next storage node, if persistence fails, data rollback is carried out, and feedback information of path updating failure is sent to the upstream of the data updating path; and after receiving the information, each upstream storage node carries out data rollback on the current updated data. According to the scheme, even if the update message processing flow fails, the consistency of the data between the survival copies / fragments can be ensured in real time.

The invention provides a distributed cluster security backup method and system oriented to a cloud-side environment, and relates to the technical field of distributed storage systems, and the method comprises the steps: obtaining cloud-side cluster data, and adaptively selecting a backup mode based on an erasure code and a multi-copy hybrid redundancy strategy; establishing a workload-aware incremental replication mechanism, and optimizing fault recovery by using idle resources; the problem of lock competition is solved by adopting a multi-copy consistency guarantee mechanism based on two-stage submission, and lock-free reading is realized based on MVCC. According to the invention, the data backup flexibility, the fault recovery efficiency and the system consistency guarantee capability are improved.

The present storage solution provides an order of operations of a computer-implemented method for performing transient failure handling with an improved application I / O resumption time for a symmetric distributed storage system; an order of operations of a computer-implemented method for performing persistent failure handling with an improved application I / O resumption time for a symmetric distributed storage system; an order of operations of a computer-implemented method for performing transient failure handling with an improved application I / O resumption time to maintain dependent write order consistency for a symmetric distributed storage system; an order of operations of a computer-implemented method for performing secondary side write Op handling to maintain dependent write order consistency for a symmetric distributed storage system; and an order of operations of a computer-implemented method for performing secondary side read Op handling to maintain dependent write order consistency for a symmetric distributed storage system in accordance with some embodiments.

The invention relates to the technical field of distributed storage system monitoring, and particularly discloses a metadata real-time monitoring method, device and equipment of a distributed storage system. The method comprises the following steps of: dynamically loading a file system mirror image to construct an initial memory metadata model by deploying an auditing service which is physically isolated from a main node and a standby node; subscribing metadata change event streams published by log nodes in real time, and incrementally updating the model after time sequence sorting and conflict detection; key dimension fields are extracted and converted into a structured format, and after batch compression, the key dimension fields are synchronized to an index engine to construct a dual-path index; and generating cross-level directory topology analysis, a user behavior portrait and storage block distribution early warning in response to query. According to the method, the problems of high time delay, metadata splitting and main node performance interference caused by offline analysis in a traditional scheme are solved, second-level updated full-quantity metadata acquisition and multi-dimensional real-time analysis are realized, and the monitoring efficiency and storage optimization capability of a massive file system are remarkably improved.

The invention discloses a block chain-based big data secure storage sharing method and system. The method comprises the following steps: firstly, preprocessing original big data, including data cleaning, desensitization and blocking operation; then the preprocessed data is encrypted, and the safety of the data in the storage and transmission process is ensured; generating a data fingerprint from the encrypted data, and storing the data fingerprint on the block chain to realize non-tampering and traceability of the data; the encrypted data is stored in a distributed storage system, so that the reliability and the expandability of data storage are improved; sharing and access control of data are realized through an intelligent contract, and only an authorized user can obtain a data key, decrypt and use the data; and finally, realizing full-life-cycle traceability of the data by using the block chain. The method has the advantages of high security, high traceability, high reliability and the like, and is suitable for security storage and sharing scenes of big data.

The embodiment of the invention provides a storage resource quota management method and a distributed storage system. In the embodiment, a token ring structure is introduced for quota management, tokens carrying quota use information are transmitted on virtual nodes in the token ring, and when any virtual node on the token ring receives the token, quota management is carried out on service operation needing to execute quota management, and the token is updated; and the updated token is transmitted to the next virtual node, so that only one virtual node can be ensured to update the same quota use information at the same time, and the problems of updating delay and inconsistency of the existing quota use information are avoided; and the problem that the quota distribution exceeds the quota limit due to the fact that multiple nodes apply for use of the quota at the same time is also avoided, so that the problems that the quota use exceeds the limit and the quota use is insufficient are fundamentally avoided, and the accuracy and reliability of storage resource quota management are improved.

The invention discloses a resource adjustment method, electronic equipment, a storage medium and a program product, and relates to the technical field of distributed storage, the weight can be dynamically adjusted according to the use condition, the emergency state and the priority of real-time IO resources of clients, the differentiated IO requirements of multiple clients can be better met, and the user experience is improved. A stable low-delay demand or a periodic burst demand can be timely and effectively responded, and the dynamic adaptability is high; by dynamically adjusting the resource allocation strategy, the bandwidth can be flexibly and reasonably allocated according to the real-time demand of each client, so that the system can operate more efficiently, and services are provided for more clients; in a distributed object storage environment and under the same hardware resource condition, the optimized IO scheduling algorithm enables the system to be more efficient when processing a client request, system response delay is reduced, the data read-write speed is increased, and therefore the performance of the whole distributed storage system is improved.

The invention discloses a hospital equipment maintenance optimization method and system based on big data analysis, and belongs to the field of equipment intelligent monitoring, and the method comprises the following steps: carrying out the preprocessing of hospital equipment operation data, and obtaining an equipment state data set of a unified format; performing real-time feature extraction on the equipment state data set by adopting a distributed computing framework to obtain a potential fault feature set; key indexes are extracted from the potential fault feature set, and equipment fault occurrence probability distribution is judged by adopting a fault prediction model; based on the fault occurrence probability distribution, analyzing an equipment priority and a maintenance scheme by adopting a decision tree algorithm to obtain an optimized maintenance decision scheme; carrying out partition recording on data generated by the maintenance decision scheme by adopting a distributed storage system, and determining a multi-layer backup structure; and extracting a data block from the multi-level backup structure, and performing consensus mechanism verification and maintenance data chain recovery on the data block by adopting a block chain technology to obtain a prediction analysis result adapted to a new environment.

The invention discloses a data reconstruction method and device, electronic equipment and a storage medium, and relates to the technical field of data storage and recovery, and the data reconstruction method comprises the steps that under the condition that a to-be-reconstructed disk exists in a distributed storage system, the redundancy of the to-be-reconstructed disk is determined based on state data of the to-be-reconstructed disk; determining a target redundancy level corresponding to the redundancy of the to-be-reconstructed disk from a plurality of preset redundancy levels; different redundancy levels correspond to different erasure code types; the erasure code types corresponding to different redundancy levels represent different combinations of data recovery rates and storage resource occupation proportions; and switching the erasure code type of the to-be-reconstructed disk to a target erasure code type corresponding to the target redundancy level, and according to the target erasure code type, transmitting the fault data block in the to-be-reconstructed disk to a specified disk for data reconstruction. The problem that data recovery performance, resource consumption and network bandwidth resource occupation cannot be dynamically balanced in the data reconstruction process in the prior art is solved.

The embodiment of the invention relates to a data processing method and device, computer equipment and a storage medium, and the method comprises the steps: carrying out the coding operation of an erasure code on first data under the condition of receiving a write-in request for the first data, and obtaining a plurality of logic units and a plurality of fragments; according to the first identifier of the logic unit and the second identifier of each fragment, writing the plurality of fragments into a storage node; and under the condition that a reading request is received, obtaining a target first identifier and a target second identifier of the second data, reading the plurality of target fragments of the second data from the plurality of storage nodes, and carrying out merging processing to obtain the second data. Therefore, the data can be divided into the fragments with the unique identifiers for storage in the writing stage, and the target fragments can be quickly positioned according to the logic addresses and the recovered data can be merged in the reading stage, so that the metadata query overhead is reduced, the reading and writing efficiency is improved, and the storage and network transmission cost is reduced; and the overall performance of the distributed storage system in a high-concurrency scene is improved.

The invention relates to the field of distributed storage, and discloses a distributed storage system, method and equipment for wind power plant data and a medium, which are used for realizing flexible adjustment of a storage strategy and meeting the storage requirement of dynamic change of a wind power plant data value. Comprising the following steps: collecting SCADA system data at edge nodes of a wind power plant, inputting an information entropy module and an access heat analysis module in parallel, fusing to generate a data value evaluation result, inputting the evaluation result into a cellular automaton engine, outputting a cellular state change instruction, summarizing collaborative strategy requests of at least two edge nodes to a regional center node, and carrying out collaborative strategy analysis on the regional center node. And a fair resource allocation scheme is output through the Sharpley value calculation module. And each edge node synchronously receives the scheme, the time sequence database controller generates a final storage instruction, and data distributed persistent storage is completed. According to the method, on the basis of the correlation between the long-term data value evaluation result and the actual use value, the entropy value and the popularity weight coefficient are optimized, and self-adaptive optimization of the system is achieved.

Techniques for implementing a scalable key value storage in a distributed storage separate the storage of the collection of data objects from the storage of the index corresponding to the collection. According to an implementation, a database service may receive a request to create a collection of data objects in an object storage. A schema may be specified for the collection. The database service may generate a unique identifier (ID) corresponding to the collection and create the collection in the object storage. The objects in the collection and the associated schema may be further written to an object table and a schema table, respectively. The database service may further validate the schema and extract one or more indexable fields from the schema. The database service may send a request to a database search service to create an index for the collection in an index storage.

The invention provides a main and standby fault switching quota management method based on a distributed storage cluster. According to the main and standby fault switching quota management method based on the distributed storage cluster, zero-error statistics and global consistency of quota data in the main and standby fault switching process can be achieved, and the high availability of a distributed storage system and the reliability of resource management are effectively guaranteed.