488 results about "Distributed memory systems" patented technology

The invention relates to the technical field of distributed storage, in particular to a high-speed distributed storage system and method, and the system comprises a file index characterization module which carries out the statistics of the access frequency and the recent access timestamp of each data block in the system, and obtains the popularity magnitude of the file data blocks; according to the method, the popularity magnitude concept is introduced on the basis of linkage statistics of the data block access frequency and the recent access time, the data blocks with remarkable service influence in the current system can be recognized, the priority processing sequence of different data blocks in the repair process is divided, and the decision-making efficiency during fault recovery is improved. After the to-be-repaired data blocks are identified, the comprehensive weight values are formed through correlation calculation with the popularity magnitude, and all the weight values are arranged in a descending order, so that dynamic grading and sequential scheduling of repair tasks are realized, and important data blocks are guaranteed to preferentially obtain recovery resource support.

The invention discloses a thematic charting processing method and system based on online data, and the method comprises the steps: carrying out the standardization processing of original data through a pre-established data processing flow, segmenting the original data into tile data, storing the tile data in a distributed storage system, and constructing a structured tile data set. And according to the structured tile data set, performing rendering processing by applying a color matching scheme, and generating a map service interface to provide a dynamic map data stream. The data is loaded to a browser end drawing environment through the map service interface, an initial drawing project file containing a drawing layout is generated, and the drawing layout comprises at least one map frame. And adding map element data into the initial cartographic engineering file and adjusting the position so as to update the cartographic engineering file. Therefore, according to the thematic charting processing method based on the online data, the data processing efficiency is improved, the cooperation threshold is reduced, and dynamic sharing is achieved.

The invention discloses a metadata prefetching method and device, a storage medium, electronic equipment and a program, and relates to the field of metadata management of a distributed storage system.The metadata prefetching method comprises the steps that a first information block corresponding to a metadata operation event in a target storage area of each node is obtained; updating the three-dimensional correlation degree between any two metadata in each node according to the first information group; determining a prefetching priority corresponding to any two pieces of metadata through the updated three-dimensional correlation degree; and generating a heat association list of the metadata in each node according to the prefetching priority, and respectively sending the heat association lists corresponding to the plurality of nodes to the client so as to indicate the client to prefetch the target metadata in the plurality of nodes according to the plurality of heat association lists. Through the method and the device, the problem of low pre-fetching accuracy caused by dependence on static path association of the metadata and poor cross-node adaptability of a pre-fetching method for the metadata in a distributed storage system is solved, and the pre-fetching accuracy is improved.

The invention discloses a resource management method and device, equipment and a medium, and relates to the technical field of storage, the resource management method comprises the following steps: carrying out performance test and data analysis on different storage architectures according to an I / O path of a distributed storage system and an asynchronous programming model, and determining processor static parameters under the different storage architectures according to the results of the performance test and the data analysis; the method comprises the following steps: collecting performance indexes of a distributed storage system through an asynchronous programming model, predicting load change conditions of the system according to the performance indexes of the distributed storage system, and dynamically adjusting processor resources according to the load change conditions of the system; managing a large-page memory pool and a cache according to the large-page memory management model and the business model type of the distributed storage system; and optimizing a memory application and release mechanism in the storage service of the distributed storage system. Through dynamic scheduling of processor resources and a large-page memory pool data sharing mechanism, balanced allocation of storage and calculation resources is realized, and the system performance and the resource utilization rate are improved.

A method and system for updating a topology on router nodes in a distributed storage system are described. The method can include obtaining, by a first service of a control plane service, topology information from an inventory data store by querying a second service of the control plane service that manages the inventory data store for the topology information. The method can further include generating, by the first service of the control plane service, a topology payload based on the obtained topology information. The method can also include sending, by the first service of the control plane service, a request comprising the topology payload to a router node.

The invention relates to the technical field of power system dispatching, in particular to a distributed storage system for source-network-load-storage collaborative dispatching, which comprises a distributed data acquisition module, a regional collaborative sensing module, a distributed energy storage evaluation module, a dispatching instruction generation module and a dispatching instruction distribution module, wherein the distributed data acquisition module is used for acquiring real-time operation data of source, network, load and storage nodes in parallel; the regional collaborative sensing module is used for calculating a real-time power deviation value; the distributed energy storage evaluation module is used for generating an energy storage adjustment instruction set; and the scheduling instruction generation module is used for generating a final refined scheduling instruction. According to the invention, through constructing a cooperative scheduling mechanism of the source network load storage full chain, accurate identification of power deviation and optimal distribution of energy storage resources are realized, and scheduling response efficiency and operation stability of the system are significantly improved.

The invention discloses a distributed storage data active migration method and system based on intelligent health prediction, and relates to the technical field of distributed storage system operation and maintaining.The method comprises the steps that multi-dimensional health state indexes of all nodes in a distributed storage cluster are periodically collected; calculating the health score of each node through a multi-time scale dynamic model based on the health state indexes; when the health score of any node is continuously lower than a preset active migration threshold value, marking the node as a high-risk node, and triggering an active migration process for all data copies on the node; constructing a multi-target optimization cost function for the to-be-migrated data, and selecting a candidate node with the lowest comprehensive cost from the healthy node pool as a migration target according to the cost function; and migrating data on the high-risk node to a selected target node at a resource-aware dynamic rate, atomically updating metadata after data migration succeeds, and then asynchronously cleaning old data copies on the high-risk node.

The invention relates to an agent content storage method and device, electronic equipment and a medium, and is applied to a distributed storage system.The method comprises the steps that in response to a session request input by a user, dialogue content corresponding to the session request is obtained, and the dialogue content is stored to first-level equipment, the first-level equipment is used for storing the real-time content generated by the intelligent agent; when it is determined that the session is ended, the session content in the first-level device is migrated to a second-level device, and the second-level device is used for storing the short-term memory content and the group sharing data of the intelligent agent; updating the content of a third-level device based on the access information of the second-level device, wherein the third-level device is used for storing long-term memory and differential memory of the intelligent agent; and the specified content in the third-level device is synchronized to a fourth-level device, and the fourth-level device is used for storing the universal knowledge base and the group unified memory so as to efficiently realize the storage of the agent data.

The invention discloses a failover method and device, electronic equipment and a storage medium, and relates to the technical field of distributed storage, and the failover method comprises the following steps: determining a first disk resource group corresponding to a first storage node and a second disk resource group corresponding to a second storage node; and establishing a double-control relationship between the first disk resource group and the second disk resource group. And after the establishment of the double-control relationship is completed, carrying out fault switching on different disk resource groups under the double-control relationship based on the first mechanism and the second mechanism. The technical problem that in a distributed storage system, efficient and safe disk resource management and data access control of a fault controller cannot be effectively achieved under a double-controller architecture is solved, and the technical effects of improving rapidness, smoothness and automatic back-switching of fault switching and remarkably improving the high availability of the distributed storage system are achieved.

The invention relates to a meter box multi-parameter prediction method and system integrated with big data processing, and the method comprises the steps: collecting the multi-dimensional original data of the operation of a meter box in real time through a built-in RTU collection module of the meter box, encrypting the original data, uploading the encrypted original data to a cloud server, and carrying out the cleaning, standardization and feature extraction, thereby obtaining structured data; storing the structured data to a distributed storage system, and indexing and compressing the data; calling the compressed structured data to construct a multi-task prediction model, and performing supervised training on the multi-task prediction model by using historical statistical data to obtain a trained prediction model; and inputting the original data acquired in real time into the trained prediction model, outputting a single-parameter prediction result, and performing multi-parameter collaborative analysis based on the single-parameter prediction result and the original data to generate a collaborative prediction result.

The invention discloses a data consistency processing method, a distributed storage system and electronic equipment, and belongs to the field of distributed data storage systems, and the data consistency processing method comprises the following steps: when any storage node in the distributed storage system receives to-be-updated data sent by a client, obtaining a corresponding data updating path by taking the storage node as a first storage node; according to the data updating path, each storage node carries out data persistence on itself after receiving the data to be updated, if persistence succeeds, the data to be updated is transmitted to the next storage node, if persistence fails, data rollback is carried out, and feedback information of path updating failure is sent to the upstream of the data updating path; and after receiving the information, each upstream storage node carries out data rollback on the current updated data. According to the scheme, even if the update message processing flow fails, the consistency of the data between the survival copies / fragments can be ensured in real time.

The invention provides a distributed cluster security backup method and system oriented to a cloud-side environment, and relates to the technical field of distributed storage systems, and the method comprises the steps: obtaining cloud-side cluster data, and adaptively selecting a backup mode based on an erasure code and a multi-copy hybrid redundancy strategy; establishing a workload-aware incremental replication mechanism, and optimizing fault recovery by using idle resources; the problem of lock competition is solved by adopting a multi-copy consistency guarantee mechanism based on two-stage submission, and lock-free reading is realized based on MVCC. According to the invention, the data backup flexibility, the fault recovery efficiency and the system consistency guarantee capability are improved.

The present storage solution provides an order of operations of a computer-implemented method for performing transient failure handling with an improved application I / O resumption time for a symmetric distributed storage system; an order of operations of a computer-implemented method for performing persistent failure handling with an improved application I / O resumption time for a symmetric distributed storage system; an order of operations of a computer-implemented method for performing transient failure handling with an improved application I / O resumption time to maintain dependent write order consistency for a symmetric distributed storage system; an order of operations of a computer-implemented method for performing secondary side write Op handling to maintain dependent write order consistency for a symmetric distributed storage system; and an order of operations of a computer-implemented method for performing secondary side read Op handling to maintain dependent write order consistency for a symmetric distributed storage system in accordance with some embodiments.

The invention relates to the technical field of distributed storage system monitoring, and particularly discloses a metadata real-time monitoring method, device and equipment of a distributed storage system. The method comprises the following steps of: dynamically loading a file system mirror image to construct an initial memory metadata model by deploying an auditing service which is physically isolated from a main node and a standby node; subscribing metadata change event streams published by log nodes in real time, and incrementally updating the model after time sequence sorting and conflict detection; key dimension fields are extracted and converted into a structured format, and after batch compression, the key dimension fields are synchronized to an index engine to construct a dual-path index; and generating cross-level directory topology analysis, a user behavior portrait and storage block distribution early warning in response to query. According to the method, the problems of high time delay, metadata splitting and main node performance interference caused by offline analysis in a traditional scheme are solved, second-level updated full-quantity metadata acquisition and multi-dimensional real-time analysis are realized, and the monitoring efficiency and storage optimization capability of a massive file system are remarkably improved.

The invention discloses a block chain-based big data secure storage sharing method and system. The method comprises the following steps: firstly, preprocessing original big data, including data cleaning, desensitization and blocking operation; then the preprocessed data is encrypted, and the safety of the data in the storage and transmission process is ensured; generating a data fingerprint from the encrypted data, and storing the data fingerprint on the block chain to realize non-tampering and traceability of the data; the encrypted data is stored in a distributed storage system, so that the reliability and the expandability of data storage are improved; sharing and access control of data are realized through an intelligent contract, and only an authorized user can obtain a data key, decrypt and use the data; and finally, realizing full-life-cycle traceability of the data by using the block chain. The method has the advantages of high security, high traceability, high reliability and the like, and is suitable for security storage and sharing scenes of big data.

The embodiment of the invention relates to a data processing method and device, computer equipment and a storage medium, and the method comprises the steps: carrying out the coding operation of an erasure code on first data under the condition of receiving a write-in request for the first data, and obtaining a plurality of logic units and a plurality of fragments; according to the first identifier of the logic unit and the second identifier of each fragment, writing the plurality of fragments into a storage node; and under the condition that a reading request is received, obtaining a target first identifier and a target second identifier of the second data, reading the plurality of target fragments of the second data from the plurality of storage nodes, and carrying out merging processing to obtain the second data. Therefore, the data can be divided into the fragments with the unique identifiers for storage in the writing stage, and the target fragments can be quickly positioned according to the logic addresses and the recovered data can be merged in the reading stage, so that the metadata query overhead is reduced, the reading and writing efficiency is improved, and the storage and network transmission cost is reduced; and the overall performance of the distributed storage system in a high-concurrency scene is improved.

The invention relates to the field of distributed storage, and discloses a distributed storage system, method and equipment for wind power plant data and a medium, which are used for realizing flexible adjustment of a storage strategy and meeting the storage requirement of dynamic change of a wind power plant data value. Comprising the following steps: collecting SCADA system data at edge nodes of a wind power plant, inputting an information entropy module and an access heat analysis module in parallel, fusing to generate a data value evaluation result, inputting the evaluation result into a cellular automaton engine, outputting a cellular state change instruction, summarizing collaborative strategy requests of at least two edge nodes to a regional center node, and carrying out collaborative strategy analysis on the regional center node. And a fair resource allocation scheme is output through the Sharpley value calculation module. And each edge node synchronously receives the scheme, the time sequence database controller generates a final storage instruction, and data distributed persistent storage is completed. According to the method, on the basis of the correlation between the long-term data value evaluation result and the actual use value, the entropy value and the popularity weight coefficient are optimized, and self-adaptive optimization of the system is achieved.

Techniques for implementing a scalable key value storage in a distributed storage separate the storage of the collection of data objects from the storage of the index corresponding to the collection. According to an implementation, a database service may receive a request to create a collection of data objects in an object storage. A schema may be specified for the collection. The database service may generate a unique identifier (ID) corresponding to the collection and create the collection in the object storage. The objects in the collection and the associated schema may be further written to an object table and a schema table, respectively. The database service may further validate the schema and extract one or more indexable fields from the schema. The database service may send a request to a database search service to create an index for the collection in an index storage.

The invention provides a main and standby fault switching quota management method based on a distributed storage cluster. According to the main and standby fault switching quota management method based on the distributed storage cluster, zero-error statistics and global consistency of quota data in the main and standby fault switching process can be achieved, and the high availability of a distributed storage system and the reliability of resource management are effectively guaranteed.

In one aspect, a method for modeling graphs on distributed storage, comprising: receiving graph data including vertices, edges, and properties; mapping each vertex to a vertex record in a distributed storage system; mapping each edge to an edge record in the distributed storage system; mapping each property to a property record in the distributed storage system; determining, for each vertex, whether a number of incident edges exceeds a maximum record size threshold; storing pointers to incident edge records directly within vertex records when the number of incident edges does not exceed the maximum record size threshold; utilizing an adjacency index to lookup incident edge records when the number of incident edges exceeds the maximum record size threshold; and executing graph traversal operations that navigate from vertex to vertex by accessing the appropriate edge records through either the stored pointers or the adjacency index.

The invention relates to the technical field of artificial intelligence, in particular to a load balancing method and device, electronic equipment, a storage medium and a program product, and the load balancing method comprises the steps: obtaining traffic characteristics of communication links between different nodes in a data block migration process of a distributed storage system; determining a load distribution mode of each communication link based on the flow characteristics of each communication link; based on the flow characteristics and the load distribution mode of each communication link, determining hotspot nodes and corresponding service priorities thereof; determining a target hotspot node based on the access frequencies, the data transmission quantities, the node loads and the corresponding service priorities of all the hotspot nodes; and based on the processing performance, the network state and the storage capacity of the node, screening out a target low-load node meeting a preset condition from the distributed storage system, and migrating data of the target hotspot node to the target low-load node. According to the invention, the network flow distribution and node load balancing state can be dynamically optimized.

This disclosure relates to decentralized integration solutions for securely conducting legal transactions and / or other types of transactions. In certain embodiments, a transaction platform includes a decentralized network integration system that interfaces with decentralized network systems, including blockchain and / or distributed storage systems, to securely conduct legal transactions and / or other types of transactions. The transaction platform can protect sensitive data associated with the transactions using techniques that combine zero-knowledge proofs (ZKP) with blockchain-based smart contract technologies. The transaction platform also can protect the integrity of transaction documentation using techniques that combine distributed storage and smart contract technologies. This disclosure also describes techniques for implementing AI-powered dispute resolution processes and tokenized ecosystems on the transaction platform.

Requests for records are performed at a record-aware distributed storage system. A request at a storage service engine for records received from a database access application has a time value identified corresponding to a state of a table and storage nodes that store the records. Requests are sent to the storage nodes to obtain the records. The storage nodes may identify rowblocks with respective record identifier ranges and time value ranges corresponding to the records and the time value for the access request. A result may be returned to the database access application based on the records received from the storage nodes.

The invention discloses a distributed storage system parameter tuning method and system based on a large language model, and belongs to the field of distributed storage system operation, maintenance and optimization, and the method comprises the steps: after all parameters are obtained, generating parameter information for each parameter based on hardware information by using an LLM-Maven model, and after the parameters irrelevant to the system performance are removed, carrying out optimization on the parameter information; verifying the legality of each parameter by using an LLM model, and prompting the LLM-Maven model to correct the parameter information when the verification is not passed; further removing immutable parameters and parameters irrelevant to system performance under each load feature according to the generated parameter information to obtain a target parameter set corresponding to each load feature, thereby constructing a sample library; after a parameter tuning request is received, carrying out feature extraction on a load in the request to obtain a target load feature; and taking the optimal configuration corresponding to the load feature most similar to the target load feature in the sample library as a current parameter tuning result. According to the invention, high-quality parameter tuning can be completed in a finite time window.

Systems and methods are provided for receiving a vector to be inserted into a set of vectors, wherein the set of vectors is stored as a plurality of leaf objects of a distributed proximity-based graph across a plurality of storage devices of a distributed object storage system, identifying a subset of the plurality of leaf objects based at least partly on a similarity of the vector to one or more vectors stored in the subset of the plurality of leaf objects, storing the vector in an insert buffer associated with the subset of the plurality of leaf objects, and in response to a vector query, generating query results comprising the vector stored in the insert buffer, and at least one of the one or more vectors stored in the subset of the plurality of leaf objects.

The invention relates to the technical field of energy data security protection, and discloses a block chain energy data security protection method, system and device based on attribute control and double-layer encryption, and a medium, and the method comprises the steps: constructing an ABAC triple policy engine based on user attributes, resource attributes and environment attributes, converting the conventional static authority management into dynamic policy verification, and carrying out the dynamic policy verification; a double-layer AES-GCM encryption algorithm is designed, the data anti-cracking capability is improved through two encryption operations, the linear time complexity is kept while the encryption security is guaranteed, and the efficiency requirement of energy data processing is considered; a dynamic key management mechanism is matched, a unique key pair is generated for each piece of data, the key leakage risk is reduced, and the problem of contradiction between safety and efficiency in a traditional encryption algorithm is solved; an on-chain and off-chain collaborative storage architecture is adopted, encrypted energy data are stored through an IPFS distributed storage system, only the hash value is subjected to on-chain verification for integrity, and capacity limitation caused by traditional block chain full-node storage is broken through.

The invention provides a method for a core fraction distributed storage system and a related product thereof. The core fraction distributed storage system comprises calculation core particles and storage core particles. The method comprises the following steps: monitoring storage related information of the core size level distributed storage system; and according to the storage related information, dynamically adjusting an access address between the calculation core grain and the storage core grain, so that the calculation core grain accesses the storage core grain nearby. Through the scheme provided by the invention, the distribution of the calculation core particles to the storage system can be dynamically adjusted, so that the calculation unit and the storage system can be tightly coupled, and the storage access delay is greatly reduced.

A request to access data is received by a storage system. The request is associated with a protocol. A determination as to whether the request is to be serviced by direct access to a hybrid data node of the storage system or by proxy access through a storage node of the storage system is made based on the protocol associated with the request. In response to determining that the request is to be serviced by proxy access, the request is serviced by retrieving the data from the hybrid data node via the storage node.

The application provides a method for load balancing of a distributed storage system in a capacity reduction scenario, comprising: obtaining original routing information of the distributed storage system; setting a slot weight for each hash slot according to a predetermined slot weight setting mode; determining a target number of slot blocks after capacity reduction of the distributed storage system; determining capacity-reduced routing information of the distributed storage system according to a predetermined load balancing algorithm based on the original routing information, the slot weight of each hash slot and the target number of slot blocks; determining capacity-reduced metadata based on the original routing information and the capacity-reduced routing information; and reassigning all hash slots managed by original nodes to reserved nodes according to the capacity-reduced metadata. The technical scheme of the application realizes dynamic load balancing of the distributed storage system in the capacity reduction process.

The invention discloses a data migration method and device, a storage medium, electronic equipment and a program product, and relates to the field of distribution.The method comprises the steps that original transaction data of a host side is split into N data fragments according to service types and time windows; encrypting each data fragment into a target data fragment; leader nodes corresponding to the target data fragments are selected from the L cluster nodes, and each target data fragment and the corresponding data version vector are sent to the corresponding leader node; performing consensus confirmation on the target data fragment received by the leader node through the leader node and the M replica nodes to obtain a consensus result of the target data fragment; and migrating the target data fragment from the host end to a storage node in a cluster. The technical problem of low consistency judgment efficiency before and after data migration caused by lack of a data version control mechanism during data migration of a distributed storage system in the prior art is solved.