815 results about "Distributed memory systems" patented technology

The invention discloses a data management method based on a distributed storage system, relates to the technical field of data management, and is used for solving the problem of storage management behavior strategy mismatch. On the basis of dynamic perception of data access behaviors in a distributed storage system, a behavior feature vector fusing an access mutation rate, periodicity and an access span is constructed, behavior pattern recognition and strategy structure generation are completed, optimal deployment and hierarchical storage of copies are achieved through node resource state perception and construction of a cost function, and the strategy structure generation efficiency is improved. And after compression and consistency configuration are executed and the strategy falls to the ground, the compressibility and the offset trend of the behavior path are analyzed, scheduling management information is extracted, and a stable execution or strategy adjustment signal is generated, so that the problem of strategy execution mismatching caused by behavior perception deficiency in the distributed storage system is reduced, and the strategy execution efficiency is improved. The strategy closed-loop control and the resource scheduling optimization under behavior driving are realized, so that the data management efficiency and the strategy adaptation capability of the distributed storage system are improved.

The invention discloses an electronic archive intelligent retrieval method and system based on a block chain, and relates to the technical field of archive retrieval, and the method comprises the steps: extracting text features of an electronic archive, and generating metadata; generating a content hash value from the electronic file original text; writing the metadata, the content hash value and the ABE access strategy into the block chain smart contract; constructing a reverse index based on the metadata semantic tag, wherein an index entry is associated with a block chain storage address; calculating a root hash and anchoring the root hash to the block chain; analyzing a keyword and a digital identity certificate in the user retrieval request; calling an intelligent contract to verify whether the user attribute accords with the ABE access strategy of the target file; retrieving an encrypted hash list matched with the file in the distributed index; acquiring the encrypted file fragments from the distributed storage system; verifying data integrity; and combining the fragments to generate a final retrieval result. The method has the advantages that through deep coupling of the block chain and attribute encryption, an electronic archive management system considering security and intelligent retrieval is constructed.

The present application discloses a data management method, apparatus and system for a distributed storage system, and an electronic device. The method comprises: traversing all storage paths in a target distributed storage cluster, and acquiring file storage information of the storage paths during traversal; on the basis of the file storage information, calculating a redundancy policy score of each storage path, and on the basis of the redundancy policy score and a first redundancy policy score threshold, determining a storage policy adapted to the storage path; when it is determined that the storage policy adapted to the storage path is an erasure code storage policy, determining a data recovery mode of the storage path on the basis of the architecture information of the storage path; and when it is determined that the storage policy adapted to the storage path is a multi-copy storage policy, determining the number of copies of the storage path on the basis of a service importance degree of the storage path and the redundancy policy score.

The invention relates to the technical field of data processing, and discloses a real-time data stream processing method and device, electronic equipment and a storage medium. The method comprises the following steps: distributing a received to-be-processed data stream to a first distributed node through a preset load balancer; fragmenting the received distribution data stream at the first distributed node according to a preset fragmentation rule, then writing the fragmented data stream into a message queue, and performing parallel processing on fragmented data in the message queue through multiple threads; storing the intermediate state data in the fragmented data processing process, generating snapshot data according to the fragmented data, and synchronizing the snapshot data to the distributed storage system; acquiring load data of the first distributed node, and dynamically updating a data flow distribution strategy of the load balancer according to the load data through the resource use prediction model; and writing a processing result of the to-be-processed data stream into persistent storage through an asynchronous batch mechanism, and returning the processing result through a preset real-time communication protocol. The method improves the data stream processing efficiency and stability.

The invention relates to the field of copyright authentication of non-perpetual culture elements, and discloses a block chain copyright authentication method for non-perpetual culture elements, which comprises the following steps: S1, performing multi-dimensional data acquisition through a distributed storage system, performing structured data acquisition and feature coding on the non-perpetual culture elements, and storing the structured data in a database; comprising the following steps: multi-modal data fusion acquisition, dynamic metadata standardization processing, and generation of unique digital identifiers and cultural feature codes; and S2, constructing a hybrid block chain architecture, constructing a hierarchical network architecture comprising a permission chain layer and an open chain layer, and realizing heterogeneous chain data synchronization through a cross-chain interoperation protocol. Through multi-modal data acquisition and dynamic metadata structured processing, the spatial form, the process time sequence and the oral context of non-residual elements are completely recorded, digital twin bodies with microscopic details and semantic association are formed, the problem of fragmentation of traditional archiving is solved, and a unique block chain identifier and two-way index mapping are combined, so that the method is simple and convenient to implement. And the originality of the cultural heritage in virtual restoration and cross-domain propagation is ensured.

The invention discloses an incidence relation establishing method, a version number generating method and device, electronic equipment, a computer readable storage medium and a computer program product, and relates to the technical field of computers.The method comprises the steps that write-in requests of N read-write nodes are obtained, each write-in request carries to-be-written data and a version number of the corresponding read-write node. And writing the to-be-written data carried by each write-in request into the corresponding logic address section, wherein the version number is used for distinguishing the to-be-written data carried by different write-in requests. Transmitting the to-be-written data to the physical address section for storage, and respectively establishing an association relationship between the physical address section of each piece of to-be-written data and the version number corresponding to each piece of to-be-written data. Different read-write requests can be distinguished by using the version number, multi-version storage of the data is realized, the corresponding data can be traced through the version number, and the processing capability of the distributed storage system for concurrent write requests is remarkably improved.

The invention relates to the field of distributed storage, in particular to an erasure code algorithm processing method for cold and hot fragmented data in a distributed storage system in a local area network or a wide area network, and solves the problems caused by using an erasure code algorithm to process cold and hot data in the prior art. The method comprises the steps of data block partitioning, a parameter strategy of data cold and hot judgment, a parameter strategy of node cold and hot judgment, data writing, an erasure code algorithm processing mechanism of different types of data, processing after erasure code algorithm processing, data cold and hot conversion processing and the like. Meanwhile, normal read-write performance is kept not affected by erasure code processing in the EC coding processing process, and the rationality of cold and hot data division in the distributed storage system is improved; layered storage and migration are performed according to cold and hot data, the read-write performance of distributed storage is considered, and the capacity utilization rate is increased; and the reliability, fault tolerance and performance of the storage system are improved.

The invention relates to the technical field of distributed storage, in particular to a high-speed distributed storage system and method, and the system comprises a file index characterization module which carries out the statistics of the access frequency and the recent access timestamp of each data block in the system, and obtains the popularity magnitude of the file data blocks; according to the method, the popularity magnitude concept is introduced on the basis of linkage statistics of the data block access frequency and the recent access time, the data blocks with remarkable service influence in the current system can be recognized, the priority processing sequence of different data blocks in the repair process is divided, and the decision-making efficiency during fault recovery is improved. After the to-be-repaired data blocks are identified, the comprehensive weight values are formed through correlation calculation with the popularity magnitude, and all the weight values are arranged in a descending order, so that dynamic grading and sequential scheduling of repair tasks are realized, and important data blocks are guaranteed to preferentially obtain recovery resource support.

The invention discloses a dynamic migration method and device of data, electronic equipment, a storage medium and a program, and relates to the technical field of distributed storage, and the method comprises the following steps: scanning N pieces of data stored in a distributed storage system through a dynamic hierarchical manager to obtain N pieces of access information; calculating a heat value corresponding to each piece of data according to the access information and a preset heat evaluation function to obtain N heat values, and dividing the N heat values according to set M heat intervals to obtain M data sets; wherein M is smaller than or equal to N, and M and N are positive integers; and carrying out dynamic migration of data storage positions on the N pieces of data based on the M data sets.

The invention provides a modularized integrated circuit unit and a distributed storage unit access method and system, and the modularized integrated circuit unit comprises at least one memory controller which sends a storage address access request to a storage unit; the at least one storage unit provides unified memory access for the storage address access request, and the storage unit comprises a local storage unit and a virtual far-end storage unit; the at least one computing unit initiates a storage address access request; and the at least one internet-on-chip is used for interleaving the storage address indicated by the storage address access request, generating interleaved storage address distribution, determining a corresponding target storage unit, and storing the target storage unit by setting a local storage unit and a virtual far-end storage unit. Uniform address access and load balancing of the distributed storage units on the modularized integrated circuit units are achieved through the address interleaving technology and the virtual storage mapping, the resource utilization rate is increased, and the performance and usability of the distributed storage system are improved.

The invention provides a trusted traffic data distributed storage system based on a main side chain, and the system comprises a traffic collection module which is used for carrying out the real-time collection of encrypted traffic data generated in a network, and carrying out the preliminary standardization processing of the encrypted traffic data, and obtaining the standardized encrypted traffic data; the traffic side chain certification module is used for carrying out integrity verification and authenticity construction on the standardized encrypted traffic data to prevent the data from being tampered or forged, so as to form verified encrypted traffic data; the traffic side chain data module is used for carrying out structured storage on the verified encrypted traffic data and reserving queriable records in a side chain; and the trusted main chain module is used for recording data abstracts, verification records, access behaviors and model voting result information on the side chain. According to the method, a block chain decentration trust mechanism and an encryption index mechanism are combined, encryption state organization and rapid retrieval of the data are achieved, and confidentiality and controllability of the data in the storage and circulation process are guaranteed.

The invention relates to the technical field of cloud computing, in particular to a cross-domain network node synchronization method and system for deploying a cloud platform. The method comprises the following steps: acquiring a cross-domain network deployment requirement of a target cloud platform and determining a node distribution strategy; designing a network topology structure supporting cross-domain communication and data synchronization; a distributed storage system supporting cross-domain data consistency is selected; realizing a cross-domain network node synchronization mechanism which comprises a heartbeat detection mechanism, an arbitration mechanism, a data synchronization mechanism and a configuration and state synchronization mechanism; and performing cross-domain network optimization and node fault switching and recovery according to a synchronization mechanism. Through a deployment strategy of multiple data centers, resource redundancy and a fault switching mechanism are realized, and the risk of a single point of fault is reduced; a network topology structure supporting cross-domain communication and data synchronization is designed, and the requirement of cross-domain virtual resource scheduling is met; and a distributed storage system supporting cross-domain data consistency is selected, so that the reliability and availability of the data are ensured.

The invention discloses a stable transmission method for server data encryption and rapid authentication, and belongs to the technical field of data security. The method comprises the following steps: dividing data into three levels of core sensitive data, important data and common data by utilizing a semantic analysis model; performing encryption protection on different levels of data by adopting a hierarchical encryption strategy; a master key is generated based on a chaotic system, a session key is generated in combination with a timestamp and a device fingerprint and is regularly alternated, and secure distribution is performed through a trusted execution environment and a block chain; multi-modal features are fused, the identity is verified through zero-knowledge proof, and the authentication strength is adjusted through dynamic risk assessment. The method is suitable for cloud computing, the Internet of Things and a distributed storage system, through deep fusion of a multi-level hybrid encryption algorithm, a dynamic key management mechanism and a self-adaptive authentication strategy, data security encryption, rapid authentication and stable transmission are achieved, and security, efficiency and adaptability are improved.

The invention discloses a thematic charting processing method and system based on online data, and the method comprises the steps: carrying out the standardization processing of original data through a pre-established data processing flow, segmenting the original data into tile data, storing the tile data in a distributed storage system, and constructing a structured tile data set. And according to the structured tile data set, performing rendering processing by applying a color matching scheme, and generating a map service interface to provide a dynamic map data stream. The data is loaded to a browser end drawing environment through the map service interface, an initial drawing project file containing a drawing layout is generated, and the drawing layout comprises at least one map frame. And adding map element data into the initial cartographic engineering file and adjusting the position so as to update the cartographic engineering file. Therefore, according to the thematic charting processing method based on the online data, the data processing efficiency is improved, the cooperation threshold is reduced, and dynamic sharing is achieved.

The invention discloses a tangelo ecological orchard data management platform, relates to the technical field of agricultural informatization and intelligent management, is used for solving the problem that decision-making efficiency is affected by data delay, and realizes real-time acquisition and classified storage of orchard environment data by arranging a sensor node network and a distributed storage system. Through data preprocessing and feature extraction, the data redundancy is reduced, and the analysis efficiency is improved; an orchard environment change trend is predicted by using a time sequence analysis algorithm, and a scientific basis is provided for resource allocation; the generation of a dynamic resource allocation scheme is realized by introducing an adaptive weight allocation mechanism and a resource priority index; and an interaction module is deployed at the orchard management terminal, so that a user can check data and adjust resource allocation in real time, and the intelligent level of orchard management is improved.

The invention provides a distributed storage system based on large model optimization, and relates to the technical field of data processing, and the system comprises a parameter analysis module which is used for obtaining parameter distribution data of a model training task; the parameter grouping module is used for performing grouping processing on the parameter distribution data; the distribution weight calculation module is used for calculating the distribution weight of each storage node in combination with the load information and the hardware performance indexes of the storage nodes; the parameter distribution module is used for distributing the parameter grouped data to different storage nodes; the dynamic monitoring module is used for monitoring access requests and load information of the storage nodes in real time in a model training or reasoning process and generating feedback information; the scheme adjusting module is used for dynamically adjusting the parameter distribution scheme according to the feedback information; the data migration module is used for executing migration operation of parameter data on the target storage node and the source storage node and updating the storage state of the parameters; the system processing capacity and efficiency can be improved.

The invention discloses a storage space management method and device, a storage medium, electronic equipment and a program product, and relates to the technical field of distributed storage, and the method comprises the steps: obtaining a plurality of physical addresses of a plurality of physical storage equipment on a server corresponding to different nodes in a distributed storage system; a plurality of physical addresses are connected in series to generate address space pools corresponding to different servers, and linked lists corresponding to the address space pools are determined; under the condition of determining that the target object initiates the logical volume creation request, determining a target number of logical addresses from the address space pool according to the linked list, and recording a mapping relationship among the logical volume, the logical addresses and a target physical address associated with the logical addresses; the storage space associated with the logical volume is managed based on the mapping relation, the storage space at least comprises two physical storage devices, and the problem that when an existing distributed storage system carries out data storage, metadata access delay is caused by data striping is solved.

The invention discloses a metadata prefetching method and device, a storage medium, electronic equipment and a program, and relates to the field of metadata management of a distributed storage system.The metadata prefetching method comprises the steps that a first information block corresponding to a metadata operation event in a target storage area of each node is obtained; updating the three-dimensional correlation degree between any two metadata in each node according to the first information group; determining a prefetching priority corresponding to any two pieces of metadata through the updated three-dimensional correlation degree; and generating a heat association list of the metadata in each node according to the prefetching priority, and respectively sending the heat association lists corresponding to the plurality of nodes to the client so as to indicate the client to prefetch the target metadata in the plurality of nodes according to the plurality of heat association lists. Through the method and the device, the problem of low pre-fetching accuracy caused by dependence on static path association of the metadata and poor cross-node adaptability of a pre-fetching method for the metadata in a distributed storage system is solved, and the pre-fetching accuracy is improved.

The invention discloses an intelligent management system for continuous education courseware resources based on big data, and relates to the technical field of education resource management. The intelligent management system for the continuous education courseware resources based on the big data comprises a data acquisition module, a data preprocessing module, a data storage module, a data management module, a personalized recommendation module and a learning path planning module, through the efficient data acquisition and preprocessing module, courseware resource data can be captured and cleaned accurately in real time, a distributed storage system is flexible in configuration, and the safety and reliability of the data are ensured. And the data management module is introduced, so that courseware resources are clearly classified and conveniently searched, and the user experience is improved. And the personalized recommendation module generates accurate resource recommendation according to user behaviors, so that the pertinence and efficiency of learning are enhanced. In addition, the learning path planning module can dynamically adjust the learning path, it is ensured that the user learns orderly according to the course dependency relationship, and the learning effect is effectively improved.

The invention discloses a data archiving processing method and system based on a shutdown system, and the method comprises the steps: carrying out the classification of data through employing a multi-dimensional classification model based on deep learning according to the data type, access frequency and business importance of the shutdown system, and obtaining a classified data set and a priority label; inputting the classified data set into a compression and encryption joint processing model based on a neural network to obtain a compressed and encrypted data packet; distributing the compressed and encrypted data packet to a distributed storage system, and generating a data storage path index by adopting an index construction algorithm based on a graph database to obtain a distributed storage index table; and according to the distributed storage index table, verifying the data integrity by using a block chain-based archiving verification technology, and generating an archiving verification report. By utilizing the embodiment of the invention, intelligent classification, effective compression and safe storage of the data in the shutdown system can be realized, and the integrity and traceability of the data are finally ensured.

Systems and methods are provided for generating an index of a set of vectors as a distributed proximity-based graph data structure comprising representative vectors corresponding to subsets of the set of vectors, storing the set of vectors across a plurality of storage devices of a distributed storage system based on the distributed proximity-based graph data structure, and loading, in response to a vector query, a plurality of subsets of the set of vectors from the distributed storage system based on the distributed proximity-based graph data structure.

In one embodiment, a computer implemented method includes storing objects in a first bucket and storing files in a second bucket of a first storage cluster of the distributed storage system, synchronously replicating data of the objects into a third mirrored bucket of a second storage cluster of the distributed storage system, synchronously replicating OSP configuration data from the first storage cluster to the second storage cluster during the synchronous replication, and providing non-disruptive operations with zero recovery time objective (RTO), and ensuring consistency between the objects in the first bucket and the objects in the third bucket for a software application that is accessing one or more objects and files using the OSP. The objects and files are accessible through an object storage protocol (OSP).

The invention discloses an intelligent engineering automatic production process quality tracing system and method, and relates to the technical field of intelligent engineering automatic production. According to the method, the data is preprocessed by using the edge computing node and the Kalman filtering algorithm, noise interference is effectively removed, and the accuracy and the real-time performance of the data are ensured; meanwhile, based on a distributed storage system and a block chain technology, data is subjected to fragmentation storage, digital signatures are added, signature consistency of data blocks is verified through an intelligent contract, a trusted data set is generated, and the problems that in a traditional tracing system, data is prone to being tampered, and safety is insufficient are solved; besides, feature extraction and quality fluctuation prediction are performed on the data by using a deep learning model and a time sequence analysis algorithm, and production parameters are dynamically optimized through a reinforcement learning model, so that early warning and process optimization of quality problems are realized, and the whole-process efficient management and accurate control capability of enterprises in a complex production environment is improved.

The invention discloses a grouping clustering method and application of a distributed storage system. The method comprises the steps that nodes in a target cluster are grouped and belong to M sub-clusters, and M is larger than or equal to 2; respectively distributing corresponding borne services for the M sub-clusters, and limiting service access among the sub-clusters; the metadata service center of the target cluster is deployed in a metadata service sub-cluster, the metadata service sub-cluster is one of M sub-clusters, and the metadata service sub-cluster comprises a main node and a standby node for operating the metadata service center of the target cluster. According to the method disclosed by the invention, the target cluster nodes are grouped, the services are allocated to the sub-clusters, and the access among the sub-clusters is limited, so that the services can be effectively isolated in the target cluster, and the security of important information data of an enterprise is ensured. The metadata service center is deployed in a metadata service sub-cluster and comprises a main node and a standby node, so that the reliability of storage service can be ensured.

The invention provides an industrial data asset distributed storage method, medium and system, and belongs to the technical field of electrical digital data processing.The method comprises the steps that firstly, manufacturing equipment operation duration data, maintenance period data, task quantity data and data generation frequency data are collected to serve as industrial original data streams, and a resource consumption calculation model is established; data features are extracted, a feature mapping matrix is constructed, resource scheduling optimization is realized through a storage optimization neural network, storage unit allocation is performed based on a deep optimization convolution kernel, dynamic evaluation and optimization adjustment are performed on data block storage positions, and finally distributed storage of industrial original data streams is realized. Through real-time calculation of the storage performance indexes and the load factors, a complete performance evaluation and optimization feedback mechanism is established, balanced distribution and efficient utilization of the storage resources are ensured, and the technical problem that in the prior art, an industrial data asset distributed storage system is difficult to achieve intelligent scheduling of the storage resources is solved.

Various embodiments of the present technology generally relate to systems and methods for providing managing object lock settings during cross-grid replication within distributed storage systems. In an example, ingestion of an object into a first grid of a distributed storage system may be detected. Responsive to detecting ingestion of the object, object lock settings for the object may be determined. Once the object lock settings are determined, the object lock settings may be validated against destination object lock settings. If the destination object lock settings are validated, cross-grid replication of the object may be initiated. During cross-grid replication, the object lock header may be provided in a replication payload transmitted from the first grid to a second grid. When the object is replicated, the destination object lock settings may be determined for the object, which may include the object lock settings as identified in the object lock header.

The invention provides a data processing method and device, equipment, a storage medium and a computer program product, and is applied to a distributed storage system.The method comprises the steps that when it is detected that a first abnormal sector appears in a first hard disk in the distributed storage system, the first abnormal sector is generated in the first hard disk; obtaining capacity information of the first abnormal sector and / or quantity information of discontinuous sectors in the first abnormal sector; under the condition that the capacity information is smaller than or equal to a first value and / or the quantity information is smaller than or equal to a second value, shielding the first abnormal sector, and searching a copy sector of the first abnormal sector in the distributed storage system; replica data in the replica sector is copied to other sectors except the first abnormal sector in the first hard disk, so that replica data of the first abnormal sector is accessed by accessing other sectors. And resource waste can be reduced.

The invention discloses a resource management method and device, equipment and a medium, and relates to the technical field of storage, the resource management method comprises the following steps: carrying out performance test and data analysis on different storage architectures according to an I / O path of a distributed storage system and an asynchronous programming model, and determining processor static parameters under the different storage architectures according to the results of the performance test and the data analysis; the method comprises the following steps: collecting performance indexes of a distributed storage system through an asynchronous programming model, predicting load change conditions of the system according to the performance indexes of the distributed storage system, and dynamically adjusting processor resources according to the load change conditions of the system; managing a large-page memory pool and a cache according to the large-page memory management model and the business model type of the distributed storage system; and optimizing a memory application and release mechanism in the storage service of the distributed storage system. Through dynamic scheduling of processor resources and a large-page memory pool data sharing mechanism, balanced allocation of storage and calculation resources is realized, and the system performance and the resource utilization rate are improved.

Disclosed in some examples are improvements to memory controllers on distributed memory systems that include a fine-grained data mover component that offloads management of memory commands accessing multiple smaller values to the memory controller. The fine-grained data mover (FGDM) may provide low host processing overhead that enables performance improvements for small task offloads. Work requests (“data mover calls”) may be sent by hosts to the FGDM without OS system calls. The FGDM is a virtually addressed data movement engine architected to transfer data at high transfer rates even during situations where the host has many small data movement requests and where the source and / or destination addresses are not memory controller friendly.

A method and system for updating a topology on router nodes in a distributed storage system are described. The method can include obtaining, by a first service of a control plane service, topology information from an inventory data store by querying a second service of the control plane service that manages the inventory data store for the topology information. The method can further include generating, by the first service of the control plane service, a topology payload based on the obtained topology information. The method can also include sending, by the first service of the control plane service, a request comprising the topology payload to a router node.