1026 results about "Distributed memory systems" patented technology

The embodiment of the invention discloses a data distributed storage method and device, and the method comprises the steps: carrying out the fragmentation processing of to-be-stored data, and obtaining an initial fragmentation result; utilizing a reinforcement learning model to perform popularity prediction on the data fragments according to the historical access data to obtain the access popularity of the data fragments; determining the number of copies based on the access popularity and importance of the data fragments, generating fragment copies of the data fragments according to the number of the copies, and storing the data fragments and the fragment copies into storage nodes according to the access popularity and node load state scores; when the loads of the storage nodes are unbalanced, data migration is carried out on the data fragments stored in the storage nodes, and when the node health state of a certain storage node is abnormal or the stored data fragments are lost, data reconstruction is carried out on the data fragments stored in the storage node. According to the method, the problems of non-uniform access load, resource waste, data loss and the like in a distributed storage system can be relieved.

The invention relates to the technical field of agricultural product quality tracing, and discloses an agricultural product quality tracing system based on a block chain technology, and the system comprises a data collection module which is used for collecting environment data, quality detection data and position information of agricultural products in the links of planting, processing, transportation, storage and the like through Internet of Things equipment; the encryption module is used for performing preliminary symmetric encryption processing on the collected data and encrypting sensitive data through a homomorphic encryption algorithm; and the intelligent contract module is used for automatically executing data storage, authority control, zero-knowledge proof verification and cross-chain data management on the block chain. According to the method, the authenticity and security of data in each link are ensured through automatic execution of data storage and authority control by using a distributed account book and a tamper-resistant characteristic of the block chain and through an intelligent contract, and meanwhile, an I PFS distributed storage system is combined with the block chain, so that the integrity of the data is further ensured, and the security of the data is improved. Therefore, all parties can share and query the tracing information trustingly.

The invention discloses a data management method based on a distributed storage system, relates to the technical field of data management, and is used for solving the problem of storage management behavior strategy mismatch. On the basis of dynamic perception of data access behaviors in a distributed storage system, a behavior feature vector fusing an access mutation rate, periodicity and an access span is constructed, behavior pattern recognition and strategy structure generation are completed, optimal deployment and hierarchical storage of copies are achieved through node resource state perception and construction of a cost function, and the strategy structure generation efficiency is improved. And after compression and consistency configuration are executed and the strategy falls to the ground, the compressibility and the offset trend of the behavior path are analyzed, scheduling management information is extracted, and a stable execution or strategy adjustment signal is generated, so that the problem of strategy execution mismatching caused by behavior perception deficiency in the distributed storage system is reduced, and the strategy execution efficiency is improved. The strategy closed-loop control and the resource scheduling optimization under behavior driving are realized, so that the data management efficiency and the strategy adaptation capability of the distributed storage system are improved.

The invention discloses a safe and rapid storage method and system for image data of an imaging department, and relates to the technical field of medical image data processing. The method comprises the following steps: 1, designing a distributed storage system, and selecting and configuring storage nodes; 2, selecting and updating an encryption algorithm, and implementing file-level and block-level encryption; 3, realizing accurate data deduplication by utilizing a machine learning technology, and selecting a proper compression algorithm; 4, constructing an index system, and establishing a data warehouse and a data lake by using big data analysis and machine learning technologies; 5, establishing an image data backup and recovery mechanism; and 6, constructing a data integrity checking and monitoring system. According to the invention, by adopting the technologies of distributed storage, encryption, data deduplication, compression, big data analysis, machine learning, backup and recovery, data integrity verification, monitoring and early warning and the like, efficient storage and safety protection of image data are realized.

The invention discloses an electronic archive intelligent retrieval method and system based on a block chain, and relates to the technical field of archive retrieval, and the method comprises the steps: extracting text features of an electronic archive, and generating metadata; generating a content hash value from the electronic file original text; writing the metadata, the content hash value and the ABE access strategy into the block chain smart contract; constructing a reverse index based on the metadata semantic tag, wherein an index entry is associated with a block chain storage address; calculating a root hash and anchoring the root hash to the block chain; analyzing a keyword and a digital identity certificate in the user retrieval request; calling an intelligent contract to verify whether the user attribute accords with the ABE access strategy of the target file; retrieving an encrypted hash list matched with the file in the distributed index; acquiring the encrypted file fragments from the distributed storage system; verifying data integrity; and combining the fragments to generate a final retrieval result. The method has the advantages that through deep coupling of the block chain and attribute encryption, an electronic archive management system considering security and intelligent retrieval is constructed.

The invention relates to the technical field of data processing, and discloses a big data-based file analysis storage division mode and a distributed storage system.The method comprises the following steps: S1, file analysis: analyzing an input big data file, and extracting a logic structure, field content and an association relationship of the big data file; s2, data block generation: dividing an analysis result into data blocks, and generating size and relevance information of the data blocks; s3, association weight generation: establishing a weight relationship between the similarity between the data blocks and the joint access frequency; and S4, storage division optimization: based on the association weight, optimizing the distribution of the data blocks among the storage nodes. According to the method, distribution of the data blocks is optimized through the synergistic effect of the association weight generation module and the dynamic decision-making module, so that the data blocks with high association degree are preferentially distributed to the same storage node, cross-node data transmission is greatly reduced, a storage strategy can be adjusted in real time, and the storage efficiency is improved. Therefore, communication delay and network resource consumption caused by cross-node access are reduced.

The present application discloses a data management method, apparatus and system for a distributed storage system, and an electronic device. The method comprises: traversing all storage paths in a target distributed storage cluster, and acquiring file storage information of the storage paths during traversal; on the basis of the file storage information, calculating a redundancy policy score of each storage path, and on the basis of the redundancy policy score and a first redundancy policy score threshold, determining a storage policy adapted to the storage path; when it is determined that the storage policy adapted to the storage path is an erasure code storage policy, determining a data recovery mode of the storage path on the basis of the architecture information of the storage path; and when it is determined that the storage policy adapted to the storage path is a multi-copy storage policy, determining the number of copies of the storage path on the basis of a service importance degree of the storage path and the redundancy policy score.

The invention discloses a big data privacy aggregation encryption method based on polymorphic fuzzy logic. The method comprises the following steps: S1, generating a to-be-encrypted data set containing privacy sensitive information; s2, constructing a fuzzy logic rule base based on polymorphic fuzzy logic; s3, generating a dynamically optimized fuzzy logic rule; s4, forming a fuzzy data set; s5, forming an optimized aggregated data block; s6, generating an encrypted data block, and generating index information corresponding to the encrypted data block; and S7, storing the encrypted data block and the index information to a distributed storage system, and dynamically controlling the access of the distributed storage system based on a data access permission rule, so that access users with different permissions can only obtain data matched with the permissions in the encrypted data block. According to the method, the privacy protection strength and efficiency are effectively improved, and a safer and more reliable solution is provided for a big data privacy protection scene.

The invention relates to the technical field of data processing, and discloses a real-time data stream processing method and device, electronic equipment and a storage medium. The method comprises the following steps: distributing a received to-be-processed data stream to a first distributed node through a preset load balancer; fragmenting the received distribution data stream at the first distributed node according to a preset fragmentation rule, then writing the fragmented data stream into a message queue, and performing parallel processing on fragmented data in the message queue through multiple threads; storing the intermediate state data in the fragmented data processing process, generating snapshot data according to the fragmented data, and synchronizing the snapshot data to the distributed storage system; acquiring load data of the first distributed node, and dynamically updating a data flow distribution strategy of the load balancer according to the load data through the resource use prediction model; and writing a processing result of the to-be-processed data stream into persistent storage through an asynchronous batch mechanism, and returning the processing result through a preset real-time communication protocol. The method improves the data stream processing efficiency and stability.

The invention relates to the field of copyright authentication of non-perpetual culture elements, and discloses a block chain copyright authentication method for non-perpetual culture elements, which comprises the following steps: S1, performing multi-dimensional data acquisition through a distributed storage system, performing structured data acquisition and feature coding on the non-perpetual culture elements, and storing the structured data in a database; comprising the following steps: multi-modal data fusion acquisition, dynamic metadata standardization processing, and generation of unique digital identifiers and cultural feature codes; and S2, constructing a hybrid block chain architecture, constructing a hierarchical network architecture comprising a permission chain layer and an open chain layer, and realizing heterogeneous chain data synchronization through a cross-chain interoperation protocol. Through multi-modal data acquisition and dynamic metadata structured processing, the spatial form, the process time sequence and the oral context of non-residual elements are completely recorded, digital twin bodies with microscopic details and semantic association are formed, the problem of fragmentation of traditional archiving is solved, and a unique block chain identifier and two-way index mapping are combined, so that the method is simple and convenient to implement. And the originality of the cultural heritage in virtual restoration and cross-domain propagation is ensured.

The invention discloses an incidence relation establishing method, a version number generating method and device, electronic equipment, a computer readable storage medium and a computer program product, and relates to the technical field of computers.The method comprises the steps that write-in requests of N read-write nodes are obtained, each write-in request carries to-be-written data and a version number of the corresponding read-write node. And writing the to-be-written data carried by each write-in request into the corresponding logic address section, wherein the version number is used for distinguishing the to-be-written data carried by different write-in requests. Transmitting the to-be-written data to the physical address section for storage, and respectively establishing an association relationship between the physical address section of each piece of to-be-written data and the version number corresponding to each piece of to-be-written data. Different read-write requests can be distinguished by using the version number, multi-version storage of the data is realized, the corresponding data can be traced through the version number, and the processing capability of the distributed storage system for concurrent write requests is remarkably improved.

Systems and methods for performing a zero-copy volume move between nodes of a distributed storage system are provided. In one example, an approach for performing a zero-copy volume move is proposed in which volume data may be maintained in place within a storage pod and need not be copied to move a given volume between the source node and the destination node. In one embodiment, metadata (e.g., a top-most physical volume block number (PVBN) of a node tree representing the volume at issue) of a write-anywhere file system is copied from the source node to the destination node. Since the storage pod is associated with a global PVBN space that is visible and accessible to all nodes of the distributed storage system, as a result of copying the top-mode PVBN of the volume to the destination node, anything below the top-most PVBN will automatically be visible to the destination node.

The invention relates to the technical field of data analysis and calculation, in particular to a distributed scheduling method based on tasks, nodes and link resources, which is applied to a solution for task scheduling and management in a large-scale distributed system. The method is particularly suitable for cloud computing, big data processing and distributed storage systems which need to process mass data and high-concurrency requests. And dynamically adjusting resource allocation by using real-time monitoring data and a prediction algorithm in combination with task priorities and resource requirements. Compared with traditional static resource allocation, a dynamic resource allocation mechanism is adopted, resource allocation can be intelligently adjusted in real time according to task requirements and node resource conditions, and therefore the resource utilization rate and the task execution efficiency are improved.

A method for reducing write latency in a distributed file system. A write request that includes a volume identifier is received at a data management subsystem deployed on a node within a distributed storage system. The data management subsystem maps the volume identifier to a file system volume and maps the file system volume to a set of logical block addresses in a logical block device in a storage management subsystem deployed on the node. The storage management subsystem maps the logical block device to a metadata object for the logical block device on the node that is used to process the write request. The mapping of the file system volume to the set of logical block addresses in the logical block device enables co-locating the metadata object with the file system volume on the node, which reduces the write latency associated with processing the write request.

The invention relates to the field of distributed storage, in particular to an erasure code algorithm processing method for cold and hot fragmented data in a distributed storage system in a local area network or a wide area network, and solves the problems caused by using an erasure code algorithm to process cold and hot data in the prior art. The method comprises the steps of data block partitioning, a parameter strategy of data cold and hot judgment, a parameter strategy of node cold and hot judgment, data writing, an erasure code algorithm processing mechanism of different types of data, processing after erasure code algorithm processing, data cold and hot conversion processing and the like. Meanwhile, normal read-write performance is kept not affected by erasure code processing in the EC coding processing process, and the rationality of cold and hot data division in the distributed storage system is improved; layered storage and migration are performed according to cold and hot data, the read-write performance of distributed storage is considered, and the capacity utilization rate is increased; and the reliability, fault tolerance and performance of the storage system are improved.

The embodiment of the invention provides a memory pool management method and device, a storage medium and an electronic device.The method comprises the steps that operation requirements of a distributed storage system are obtained, and large-page memory division is conducted on storage resources associated with the distributed storage system according to the operation requirements to obtain a plurality of target memory pools; wherein each target memory pool in the plurality of target memory pools corresponds to one type of memory object type, and the one type of memory object type at least comprises one memory object; recording historical data of using the plurality of target memory pools in a first preset period; constructing an intelligent prediction model through the historical data and a preset initial prediction model, and predicting an adjustment demand in a second preset period by using the intelligent prediction model and the current state data of the distributed storage system; and dynamically managing the plurality of target memory pools based on the adjustment demand. The problems that memory allocation is high in overhead and low in efficiency, and dynamic management and optimization of memory resources cannot be achieved are solved.

The invention relates to the technical field of distributed data storage, and discloses a data index construction method for distributed data storage, comprising the following steps: constructing a hypergraph model of a distributed storage system through a high-order graph theory, the hypergraph model comprising a plurality of storage nodes, a plurality of data blocks and a plurality of query requests, the method comprises the following steps: performing load balancing optimization on storage nodes and a query request on the basis of a minimum cut theorem in a high-order graph model, mapping data blocks and the storage nodes to a consistent hash ring according to a consistent hash algorithm, reducing data migration through a virtual node mechanism, storing the data blocks on a plurality of storage nodes in a redundant manner by adopting a redundant coding method, and storing the data blocks on the storage nodes in a redundant manner. And a load distribution strategy of the storage nodes is adjusted through a game theory model, and load adjustment is optimized by adopting an optimal control method. The method can ensure high availability, expansibility and reliability of the system while improving the efficiency of the data storage system, and is suitable for a large-scale distributed data storage system.

The invention relates to the technical field of distributed storage, in particular to a high-speed distributed storage system and method, and the system comprises a file index characterization module which carries out the statistics of the access frequency and the recent access timestamp of each data block in the system, and obtains the popularity magnitude of the file data blocks; according to the method, the popularity magnitude concept is introduced on the basis of linkage statistics of the data block access frequency and the recent access time, the data blocks with remarkable service influence in the current system can be recognized, the priority processing sequence of different data blocks in the repair process is divided, and the decision-making efficiency during fault recovery is improved. After the to-be-repaired data blocks are identified, the comprehensive weight values are formed through correlation calculation with the popularity magnitude, and all the weight values are arranged in a descending order, so that dynamic grading and sequential scheduling of repair tasks are realized, and important data blocks are guaranteed to preferentially obtain recovery resource support.

The invention discloses a dynamic migration method and device of data, electronic equipment, a storage medium and a program, and relates to the technical field of distributed storage, and the method comprises the following steps: scanning N pieces of data stored in a distributed storage system through a dynamic hierarchical manager to obtain N pieces of access information; calculating a heat value corresponding to each piece of data according to the access information and a preset heat evaluation function to obtain N heat values, and dividing the N heat values according to set M heat intervals to obtain M data sets; wherein M is smaller than or equal to N, and M and N are positive integers; and carrying out dynamic migration of data storage positions on the N pieces of data based on the M data sets.

The invention discloses a solid state disk management method and device, equipment, a medium and a product, which are applied to the technical field of storage and are used for solving the problem of data loss caused by difficult realization of timely and accurate solid state disk anomaly detection in the traditional technology, and the method is applied to a distributed storage system deployed with a service layer and a storage layer. The storage layer is deployed with a plurality of object storage devices, each object storage device is deployed with a plurality of solid state disks, and the method comprises the following steps: monitoring a service layer to determine whether the service layer is abnormal or not; if yes, determining an abnormal solid state disk and a solid state disk exception type in a storage layer; the exception type of the solid-state disk comprises exception of the disk and / or expiration of the service life of the disk; when the exception type of the solid state disk is the exception of the solid state disk, performing fault detection on the exceptional solid state disk by utilizing a preset fault detection strategy to obtain a fault detection result; and determining a target protection strategy by using the solid state disk exception type and the fault detection result so as to perform redundancy protection on the exceptional solid state disk.

The invention provides a modularized integrated circuit unit and a distributed storage unit access method and system, and the modularized integrated circuit unit comprises at least one memory controller which sends a storage address access request to a storage unit; the at least one storage unit provides unified memory access for the storage address access request, and the storage unit comprises a local storage unit and a virtual far-end storage unit; the at least one computing unit initiates a storage address access request; and the at least one internet-on-chip is used for interleaving the storage address indicated by the storage address access request, generating interleaved storage address distribution, determining a corresponding target storage unit, and storing the target storage unit by setting a local storage unit and a virtual far-end storage unit. Uniform address access and load balancing of the distributed storage units on the modularized integrated circuit units are achieved through the address interleaving technology and the virtual storage mapping, the resource utilization rate is increased, and the performance and usability of the distributed storage system are improved.

The invention provides a trusted traffic data distributed storage system based on a main side chain, and the system comprises a traffic collection module which is used for carrying out the real-time collection of encrypted traffic data generated in a network, and carrying out the preliminary standardization processing of the encrypted traffic data, and obtaining the standardized encrypted traffic data; the traffic side chain certification module is used for carrying out integrity verification and authenticity construction on the standardized encrypted traffic data to prevent the data from being tampered or forged, so as to form verified encrypted traffic data; the traffic side chain data module is used for carrying out structured storage on the verified encrypted traffic data and reserving queriable records in a side chain; and the trusted main chain module is used for recording data abstracts, verification records, access behaviors and model voting result information on the side chain. According to the method, a block chain decentration trust mechanism and an encryption index mechanism are combined, encryption state organization and rapid retrieval of the data are achieved, and confidentiality and controllability of the data in the storage and circulation process are guaranteed.

The invention relates to the technical field of cloud computing, in particular to a cross-domain network node synchronization method and system for deploying a cloud platform. The method comprises the following steps: acquiring a cross-domain network deployment requirement of a target cloud platform and determining a node distribution strategy; designing a network topology structure supporting cross-domain communication and data synchronization; a distributed storage system supporting cross-domain data consistency is selected; realizing a cross-domain network node synchronization mechanism which comprises a heartbeat detection mechanism, an arbitration mechanism, a data synchronization mechanism and a configuration and state synchronization mechanism; and performing cross-domain network optimization and node fault switching and recovery according to a synchronization mechanism. Through a deployment strategy of multiple data centers, resource redundancy and a fault switching mechanism are realized, and the risk of a single point of fault is reduced; a network topology structure supporting cross-domain communication and data synchronization is designed, and the requirement of cross-domain virtual resource scheduling is met; and a distributed storage system supporting cross-domain data consistency is selected, so that the reliability and availability of the data are ensured.

The invention discloses a stable transmission method for server data encryption and rapid authentication, and belongs to the technical field of data security. The method comprises the following steps: dividing data into three levels of core sensitive data, important data and common data by utilizing a semantic analysis model; performing encryption protection on different levels of data by adopting a hierarchical encryption strategy; a master key is generated based on a chaotic system, a session key is generated in combination with a timestamp and a device fingerprint and is regularly alternated, and secure distribution is performed through a trusted execution environment and a block chain; multi-modal features are fused, the identity is verified through zero-knowledge proof, and the authentication strength is adjusted through dynamic risk assessment. The method is suitable for cloud computing, the Internet of Things and a distributed storage system, through deep fusion of a multi-level hybrid encryption algorithm, a dynamic key management mechanism and a self-adaptive authentication strategy, data security encryption, rapid authentication and stable transmission are achieved, and security, efficiency and adaptability are improved.

The invention discloses a distributed storage system, a space allocation method and device, electronic equipment and a nonvolatile storage medium, and is applied to the technical field of storage. The method comprises the steps that a storage space is divided into a plurality of storage areas with the same storage capacity and continuous addresses in advance, and the corresponding relation between storage bitmaps and the storage addresses of corresponding storage units is persisted to the storage space; the bits of the storage bitmap indicate whether the state of the corresponding storage unit is an idle state or an allocated state. And in a storage node starting process, determining a target storage area in an idle state in the storage space according to the corresponding relationship, and loading bitmap information corresponding to the target storage area into a memory. According to the method and the device, the problems of excessive memory occupation and low service recovery speed in related technologies can be solved, less memory can be occupied when the PB-level space is managed, meanwhile, the loading time during node restarting is shortened, and the recovery speed of node operation services is effectively improved.

The invention discloses a power grid construction subpackage management and control method based on big data, and the method comprises the following steps: collecting the data of a power grid construction site in real time, carrying out the preprocessing, and carrying out the storage through a distributed storage system; generating a comprehensive evaluation report by using the residual neural network and the graph convolutional network; adopting a particle swarm optimization algorithm to carry out subpackage provider selection and task allocation; predicting the construction progress through a long short-term memory network; performing image analysis on the video stream of the construction site by using a residual neural network and a single-time multi-frame detection algorithm; a Bayesian network and a multi-layer perceptron are used to predict construction risks, an isolated forest algorithm is used to carry out abnormal value detection on data, and a risk early warning report is generated and notified to related personnel. According to the invention, big data, deep learning and reinforcement learning technologies are fully combined, the power grid construction subpackage management and control method based on big data is described in detail, and the method has the advantages of high construction efficiency, accurate risk prediction and resource optimization.

The invention discloses a thematic charting processing method and system based on online data, and the method comprises the steps: carrying out the standardization processing of original data through a pre-established data processing flow, segmenting the original data into tile data, storing the tile data in a distributed storage system, and constructing a structured tile data set. And according to the structured tile data set, performing rendering processing by applying a color matching scheme, and generating a map service interface to provide a dynamic map data stream. The data is loaded to a browser end drawing environment through the map service interface, an initial drawing project file containing a drawing layout is generated, and the drawing layout comprises at least one map frame. And adding map element data into the initial cartographic engineering file and adjusting the position so as to update the cartographic engineering file. Therefore, according to the thematic charting processing method based on the online data, the data processing efficiency is improved, the cooperation threshold is reduced, and dynamic sharing is achieved.

The invention provides an image data processing and storing method based on a meta universe game, which comprises the following steps of: integrating and primarily processing image data to obtain an original image data set; a graphic database is designed based on a distributed storage system and by adopting a cold and hot data hierarchical storage strategy; processing the original image data set by utilizing a self-adaptive compression algorithm, a multi-thread asynchronous loading mode, GPU (Graphics Processing Unit) parallel computing, a progressive rendering technology, a physical basic rendering technology and illumination computing to obtain a standard image data set; a virtualization technology is utilized to simulate various hardware configurations, and a plurality of APIs are combined to carry out cross-platform compatible management of image data; and automatically updating and optimizing the content of the image data according to the monitoring result of the image data. According to the method, efficient storage and processing of the image data can be realized, the quality and compatibility of the image data are enhanced, the game performance and the user experience are remarkably improved, and development and popularization of the virtual world are promoted.

The invention discloses a tangelo ecological orchard data management platform, relates to the technical field of agricultural informatization and intelligent management, is used for solving the problem that decision-making efficiency is affected by data delay, and realizes real-time acquisition and classified storage of orchard environment data by arranging a sensor node network and a distributed storage system. Through data preprocessing and feature extraction, the data redundancy is reduced, and the analysis efficiency is improved; an orchard environment change trend is predicted by using a time sequence analysis algorithm, and a scientific basis is provided for resource allocation; the generation of a dynamic resource allocation scheme is realized by introducing an adaptive weight allocation mechanism and a resource priority index; and an interaction module is deployed at the orchard management terminal, so that a user can check data and adjust resource allocation in real time, and the intelligent level of orchard management is improved.

The invention provides a distributed storage system based on large model optimization, and relates to the technical field of data processing, and the system comprises a parameter analysis module which is used for obtaining parameter distribution data of a model training task; the parameter grouping module is used for performing grouping processing on the parameter distribution data; the distribution weight calculation module is used for calculating the distribution weight of each storage node in combination with the load information and the hardware performance indexes of the storage nodes; the parameter distribution module is used for distributing the parameter grouped data to different storage nodes; the dynamic monitoring module is used for monitoring access requests and load information of the storage nodes in real time in a model training or reasoning process and generating feedback information; the scheme adjusting module is used for dynamically adjusting the parameter distribution scheme according to the feedback information; the data migration module is used for executing migration operation of parameter data on the target storage node and the source storage node and updating the storage state of the parameters; the system processing capacity and efficiency can be improved.