164 results about "Distributed memory" patented technology

The invention discloses a large model training fault recovery method based on distributed memory management, and the method comprises the steps: obtaining a model state at a current moment when judging that a check point storage service needs to be carried out in a current training round, serializing the obtained model state into a storable format, and generating check point data; then transmitting the check point data to a host side memory from a device side memory in the node; after the transmission is finished, continuing to train the task, and storing asynchronous check points at the same time; the asynchronous check point storage comprises data distributed memory backup and data persistence to a disk; and when the training task goes wrong, task recovery is carried out through check point data. When the fault is recovered, the backup data is preferentially read from the memory, and the delay caused by the I / O bottleneck of the disk can be obviously reduced, so that the training pause or the efficiency reduction caused by the operation of storing or loading the check point data is reduced, the efficient utilization of the GPU or other computing resources is kept, and the continuity and the stability of large-scale model training are improved.

This application generally relates to memory provisioning and particularly describes systems and methods for distributed memory pools. For example, a computing / memory device in a distributed system may be configured to execute application logics using local cache for data maintained in non-local memory in another computing / memory device of the distributed system. The computing / memory device may further be configured to provide memory for other applications executed on other computing / memory devices of the distributed system. The local and non-local memory allocation for the applications in the computing / memory devices of may be dynamically and adaptively adjusted according to various system parameters and indicators.

The invention provides a parallel computing method for supporting real-time data stream processing in parallel programming, which comprises the following steps of: dividing a real-time data stream into fragments through a parallel pipeline, and filtering specified fields by different threads to improve the efficiency; after filtering is completed, performing local aggregation on a result by using a local aggregation function of MapReduce to reduce the data volume; a distributed memory database based on Redis Cluster is established, and data are stored in a fragmented mode according to a time window; constructing a long-short-term memory recurrent neural network model, and performing parallel training through a parameter server architecture; if the long short-term memory recurrent neural network model needs incremental learning, a parameter interface is reserved for iterative updating, and repeated training is avoided; after training is completed, the model is deployed to a GPU server, and real-time data flow is predicted; through a task scheduler of a distributed memory computing framework, cooperation between threads and processes is automatically managed, and programming work is simplified.

The invention provides a concurrent debugging method and system based on sandbox-LLM (Language Model) session bidirectional binding. The method comprises the following steps: distributing a global unique session identifier for each LLM session; writing the globally unique session identifier and the sandbox instance identifier into a distributed memory database to form a bidirectional binding table, and marking the initial state as a binding state; realizing unvarnished transmission of a full-link global unique session identifier through an environment variable and an HTTP / gRPC header field; during sandbox thermal migration, the old instance atomically updates a binding table entry state into a migration state through a distributed mutual exclusion lock, and the new instance is written into the binding state after being preheated within a preset time; the log agent identifies an aggregation index according to a sandbox instance, collects average generation delay according to a preset frequency, accumulates a data collection number and GPU video memory occupation; and periodically polling the binding table, and cleaning entries which exceed a preset threshold value of the distributed mutual exclusion lock and are in a recovery state.

The embodiment of the invention discloses a solid state disk read-write method and a computer readable storage medium. The method is applied to a read-write module in a kernel module of an operating system. The read-write module is used for maintaining a continuous virtual memory page space and a distributed memory page distributor, the virtual memory page space is composed of a plurality of virtual memory pages, and each virtual memory page is in one-to-one correspondence with a solid state disk page; the distributed memory page distributor is composed of a plurality of working threads, and each working thread manages a local memory pool composed of physical memory pages; the method comprises the following steps: analyzing an obtained read-write instruction to determine to-be-processed target data and a read-write demand; positioning a corresponding virtual memory page and a physical memory page according to the target data address; and processing the target data in the solid state disk page and / or the physical memory page according to the read-write demand so as to complete the read-write operation of the target data. The file reading and writing efficiency of the solid state disk can be improved.

The invention relates to a remote sensing image data processing method and system based on a Spark architecture, and belongs to the technical field of remote sensing image data processing. The method comprises the following steps: converting a collected original file into a COG format, constructing a hybrid index through a spatial dimension reduction technology, extracting metadata and establishing a metadatabase; based on the hybrid index, quickly positioning a target tile through spatial query, reading a data block, and constructing a TileLayerRDD of a spatial partition in a Spark cluster; and dynamically assembling and calculating the directed acyclic graph according to the algorithm warehouse template, and submitting the directed acyclic graph to a Spark cluster for distributed memory calculation. According to the method, rapid retrieval is achieved through the COG format and the mixed index, pre-slicing is replaced with Spark distributed computing, the storage pressure is greatly reduced, and on-demand real-time analysis and high-concurrency online service of mass remote sensing data are finally achieved through micro-service and elastic resource scheduling.

A computer-implemented method implemented in a blockchain network is described. Validation nodes receive data regarding a newly mined block comprising a plurality of transactions and send a delete request to a distributed memory pool to delete the plurality of transactions from the distributed memory pool. Nodes storing the distributed memory pool store a plurality of transactions, the plurality of transactions forming at least part of a distributed memory pool of transactions waiting to be mined into a block of a blockchain. The computer-implemented method further comprises receiving a delete request from a validation node of the blockchain network, the delete request identifying one or more transactions which have been included in a newly mined block, the delete request indicating that the one or more transactions should be deleted from the distributed memory pool.

Systems and techniques include identifying a network layer for performing a memory operation, identifying a subset of a plurality of configuration bit clusters of a non-volatile distributed memory that are mapped to the identified network layer using a cluster mapping, in response to identifying the subset of the plurality of configuration bit clusters, activating the subset of the plurality of configuration bit clusters, loading network component data from the subset of the plurality of configuration bit clusters into a local buffer, and applying the network component data to the network layer for performing the memory operation.

The invention discloses a memory access device, a processor and a memory access method, and belongs to the technical field of computers. The memory access device comprises a scheduling module which is configured to distribute memory access instructions in an instruction queue to a plurality of assembly lines of the memory access device; the first cache module is configured to cache the first mapping relation; the control module is configured to query the page table and update the second mapping relation in the second cache module according to the memory access instruction of the virtual machine monitor if the memory access instruction currently processed by the plurality of assembly lines does not hit the first mapping relation and the memory access instruction is the memory access instruction of the virtual machine monitor; and the second cache module is configured to cache the second mapping relation. On the basis of not increasing the complexity of hardware design, the resource utilization rate in the execution process of the memory access instruction of the virtual machine monitor can be improved.

A compute node (400) for a computer, a computer (600) with a distributed memory as main memory assembled from the compute nodes, and a memory access method (800) for such computers are disclosed. The compute node (400) comprises at least one compute core (410) operating on information organized and stored in memory objects (10). Memory objects are subdivided into fragments (12) which are referenceable by the at least one compute core. A memory object management unit (420) of the compute node operatively couples the at least one compute core (410) to the distributed memory to access fragments (12) from respective memory locations in the distributed memory, which are accessible through corresponding physical addresses. The memory object management unit (420) is configured to calculate the physical addresses as a function of the fragment offset in a logical address space (11) associated with and a unique identifier (OID) assigned to the memory object containing referenced fragments.

The invention discloses a data processing method and device, equipment and a medium, and relates to the technical field of computers, and the method comprises the steps: receiving real-time data streams sent by a plurality of clients through a target server cluster; wherein the target server cluster comprises an asynchronous socket server based on a non-blocking network, and the target server cluster realizes event-driven network data processing based on a reactor mode; according to the real-time load of each partition in a target message queue, distributing the received real-time data flow to the corresponding partition in the target message queue; cleaning the real-time data flow of each partition in the target message queue by using a target rule engine with a hot loading function to obtain cleaned data; and storing the cleaned data to a distributed memory database cluster. Therefore, resource tension and response delay caused by connection congestion are reduced, and data consistency and service continuity are guaranteed.

The invention discloses a database distributed memory management method and device, equipment and a storage medium, and relates to the field of databases, and the method comprises the following steps: in a target database, constructing a distributed memory pooling framework by taking a database instance as a node, and under the distributed memory pooling framework, creating a mapping table by utilizing a memory mapping technology; accessing the shared cache page based on the mapping table among the nodes, and dynamically adjusting the distribution of the private cache page through a preset adaptive load monitoring algorithm and a preset load weight determination formula; and when a data write-in request of the node is obtained, controlling the request flow based on a token bucket algorithm, and executing a write operation on to-be-modified data located in the shared cache page in the node by using a timestamp-based optimistic lock mechanism. Therefore, the management efficiency of the distributed memory in the database can be improved.

Intelligent secure methods, processes, systems, and apparatus are disclosed for controlling and monitoring volatile memories in third-party vendor systems, such as an ATM or PoS, providing a solution to track and integrate memory components into the monitoring system of a customer's financial institution using scanning procedures, homomorphic encryption, and blockchain transactions to establish trust and ensure confidentiality by orchestrating the coupling of distributed memory storage located on the third-party vendor systems and / or hardware by leveraging PCCO (Parity Check Control Object) and DIP (Dependency Inversion Protocol).

The invention belongs to the technical field of stream computing data processing, and particularly discloses a stream computing data processing system based on kappa architecture construction, which comprises an event acquisition unit, an event processing unit, an event storage unit, an event computing unit and a management platform, through cooperation of the event collection unit, the event processing unit, the event storage unit and the event calculation unit, collection and cleaning of events are completed through a kafka + flink component, real-time calculation of the events is completed through a streaming big data calculation engine, real-time index storage is performed through a distributed memory technology, and real-time calculation of the events is completed. The use of components is ensured through a distributed cluster mechanism, and real-time calculation data is timely returned to a batch library, so that a Kappa architecture can be applied to data platform establishment, the streaming data value is fully mined, the business operation efficiency and the risk monitoring timeliness are effectively improved, and by integrating the streaming data and the batch data, the risk monitoring efficiency is improved. And the batch processing operation efficiency of the data platform is improved.

The invention relates to a bus track offset storage and query method based on a distributed memory index, and belongs to the technical field of track semantic storage and track search. The method comprises the following steps: firstly, acquiring urban line information and bus GPS track data, and respectively calculating grid sequences corresponding to a bus predetermined line and a bus actual GPS track; and further calculating the similarity between the actual GPS track of each bus and the predetermined route, and obtaining the route offset information of each bus. All the offset information is re-partitioned according to a time-grid mode by using an RDD elastic distributed data set of the Spark, and a global index is formed. In each partition, multi-level division is carried out according to attributes such as dates, lines and license plate numbers, and an auxiliary filtering vector is established in each level, so that a local index is formed. And finally, defining a spatio-temporal query interface, and supporting multi-query condition combination for query. According to the method, line deviation retrieval with semantic information can be quickly and efficiently executed in a super-large-scale GPS data set.

Systems and techniques include identifying a network layer for performing a memory operation, identifying a subset of a plurality of configuration bit clusters of a non-volatile distributed memory that are mapped to the identified network layer using a cluster mapping, in response to identifying the subset of the plurality of configuration bit clusters, activating the subset of the plurality of configuration bit clusters, loading network component data from the subset of the plurality of configuration bit clusters into a local buffer, and applying the network component data to the network layer for performing the memory operation.

The invention discloses a method, a system and a medium for expanding and accelerating communication between microkernel processes by using memory labels, and the method comprises the following steps: distributing memories for application programs and system services, and labeling the distributed memories; realizing an isolated memory domain and an execution domain based on the label; inter-process communication is carried out in a library operating system or kernel module form by utilizing memory mark extension characteristics; and cross-service interaction is realized through the fast channel. According to the invention, the delay of inter-process communication of the microkernel is obviously reduced, and meanwhile, effective isolation and security are ensured.

The invention belongs to the technical field of chip algorithms, and particularly relates to a circular array beam forming method and system based on a domestic FPGA. Comprising the following steps: receiving multiple paths of array element signals, and performing frequency mixing and filtering to obtain baseband I / Q data; the base band I / Q data is unpacked to be distributed and stored in a plurality of distributed memories in an FPGA (Field Programmable Gate Array) processing unit; the method comprises the following steps: reading baseband I / Q data of a plurality of channels and corresponding phase shift coefficients in parallel, and carrying out multiplication, accumulation, summation and modulo operation to generate data of a beam point; repeating the calculation step of generating the beam point data by updating the phase shift coefficient so as to generate beam domain data; reordering the calculated sequence of the beam domain data, and adjusting the output sequence of the beam domain data; the method has the advantages of being high in processing speed, high in anti-noise capacity and the like on the basis of meeting domestic substitution.

The invention provides a two-dimensional code production method and system, electronic equipment and a medium, and relates to the technical field of two-dimensional code production, and the method comprises the steps: obtaining a character string constraint, and obtaining a character string set through a random character string generation plan under the character string constraint; a MySQL horizontal segmentation strategy is introduced to distribute and store the character string set to a distributed memory, and the distributed memory comprises a first memory and a second memory; writing the first character string group in the first memory and the second character string group in the second memory into a MySQL database in parallel; and calling a two-dimensional code generation library to convert the target character string in the MySQL database into a target two-dimensional code. According to the method and the device, the problem of low generation speed of an existing traditional two-dimensional code production scheme can be solved, and the technical effect of remarkably improving the generation and storage efficiency of the two-dimensional code is achieved.

An electric vehicle charging and discharging operation regulation platform, comprising: a data collection cluster, which is configured to receive and parse collected data sent by an external system, wherein the external system comprises multiple terminals, an Internet-of-Vehicles platform and a power grid dispatching platform; a data processing cluster, which is configured to perform processing analysis and periodic sampling storage on real-time remote signaling / telemetering data; a data storage cluster, which has a distributed memory database constructed therein, and is configured to implement the large-capacity storage of historical data and high-concurrency access to the historical data; and a micro-service cluster, which is configured to implement, on the basis of data stored in the data storage cluster, a service application micro-service and a platform micro-service by means of a micro-service architecture.

A high-speed parallel GRAND-CRC decoding method for functional safety communication is presented. M A round-robin noise error mode allocation scheme for each memory cell, without affecting error correction performance, provides... N The system adds an all-zero error mode to the set E of noise error modes and allocates noise error modes to each memory unit in turn. A high-speed parallel decoder architecture design scheme for GRAND-CRC is proposed, including a distributed memory design strategy and the advantage of unrestricted parallelism. The parallelism can be easily set according to hardware resources, and the decoder throughput increases linearly with the parallelism. The system uses a corrector as the basis for logical judgment, and the corrector signal processing unit triggers the correct decoding output. Its features include different processing methods in three cases, especially when contention occurs, selecting the guessed sequence corresponding to the corrector with the smallest sequence number as the decoding output.

A device for implementing sparse distributed memory may include first circuitry having a plurality of cells arranged in first subsets. The first circuitry may be configured to receive a first input vector for a write operation and calculate a similarity between the first subsets of the first circuitry and the first input vector. The device may include second circuitry coupled to the first circuitry and configured to output a first activation signal. The device may include third circuitry coupled to the second circuitry and having cells arranged in second subsets. The third circuitry may be configured to receive the first activation signal and selectively activate one or more first selected subsets of the second subsets in response to the first activation signal by incrementing or decrementing corresponding values of the cells of the one or more first selected subsets based on the first input vector.

The invention discloses a bidirectional replication consistency checking and repairing method based on a record source identifier, particularly relates to the technical field of data consistency of a distributed memory database, and is used for solving the problem that a bidirectional replication cluster is difficult to accurately identify a record source, missing and conflicts in a non-stop scene, so that the reliability of the bidirectional replication cluster is improved. Therefore, the problems of difficulty in consistency verification, high possibility of false difference and repeated modification are solved. A deletion mark, a lock mark and a source mark are coded in a record control area in a centralized manner, a primary key field and a source pseudo column are used as anchor points, and an idempotent identifier constructed based on a primary key and a version field, a fragment comparison process divided according to a verification window and a primary key range and an automatic repair decision oriented to missing records and records with inconsistent fields are combined; therefore, the real difference can still be accurately recognized under the condition of bidirectional replication continuous operation, replication loopback and false inconsistency are avoided, and the consistency of the business data of the memory database clusters on the two sides is kept on the premise that business writing is not stopped.

A system and method for configuring bits using distributed memory in an artificial neural network is provided. Systems and techniques include identifying a network layer for performing memory operations; identifying a subset of a plurality of configuration bit clusters of the volatile distributed memory, the configuration bit clusters of the subset being mapped to the identified network layer using a cluster mapping; in response to identifying the subset of the plurality of configuration bit clusters, initiating the subset of the plurality of configuration bit clusters; loading network component data from the subset of the plurality of configuration bit clusters into a local buffer; and applying the network component data to the network layer for performing the memory operation.

The invention discloses a mathematical material storage and query management system based on big data, and relates to the technical field of data management. The system comprises a database unit, a front-end login unit and a rear-end service unit, wherein the database unit is used for respectively storing electronic textbooks and entity textbooks in online and offline forms; the electronic textbook is stored in a distributed memory, and the entity textbook is stored in an intelligent bookcase; the front-end login unit is used for logging in the use interface according to the account; the back-end service unit is used for performing hierarchical retrieval in a currently logged-in use interface according to keywords, and performing visual display through a popped data monitoring interface after the retrieval is completed; the data monitoring interface comprises all electronic textbook names related to the keywords, corresponding textbook content links, all entity textbook names related to the keywords and corresponding cabinet body positions. The application and management efficiency of the textbook can be improved.

A system for administering regionally-based data storage and control is provided. The system may include a computer processor and a distributed memory. The distributed memory may be distributed across a plurality of regions. The distributed memory may provide data storage. The system may build a plurality of artificial intelligence (AI) models. Each of the models may correspond to a set of data rules. Each of the data rules governs the storage of data in each of the plurality of regions. The system may further create, or otherwise access, a plurality of data keys. Each of the data keys preferably corresponds to one of the regions. The system preferably centralizes the location of the plurality of data keys in a central location such that control of the data in the plurality of regions can be administered by the processor, via the plurality of data keys, from the central location.

The invention relates to the technical field of communication, and discloses a memory management method and device for data forwarding, equipment and a storage medium. The method comprises the following steps: dividing a memory pool into a memory common area and a memory expansion area; wherein the memory common area is of a fixed size, and the memory expansion area is of a dynamic size; when a message is forwarded, distributing memory resources of the memory common area or the memory expansion area to the outside according to a current memory resource usage amount pointer so as to map and store the message; wherein each memory resource distributed to the outside by the memory common area and the memory expansion area has a corresponding mark; and when forwarding of the message is finished, returning the memory resource storing the message to a region corresponding to a memory pool based on the mark corresponding to the memory resource. Through the method and the device, the technical problem that in the prior art, in order to ensure that each resource in the memory pool is sequential and continuous, only a complex algorithm can be added is solved.

A method and a system for subdividing, distributing and storing data associated with a subject in a plurality of distributed memories of a telecommunications network. The system includes a telecommunications network that includes a plurality of network nodes with decentralized architecture. At least one server component connected to the telecommunications network includes a plurality of fragment storage nodes connected to the telecommunications network. Each fragment storage node includes at least one respective non-volatile memory configured to store at least part of a plurality of portions of the data.

Techniques and computer-implemented methods are presented to be implemented on a distributed mempool network (DMP) implementing a distributed hash table (DHT). The method includes: receiving, at a mempool node of a DMP, a request to update routing information of the mempool node; initializing a set of random walks within a key space of the DHT, the set of random walks used for generating a set of key-value records; selecting a particular iteration of a long-distance table of a set of long-distance tables; selecting, from the particular iteration of the long-distance table, a pointer within a range between identification information and a key; and transmitting a query to another mempool node associated with the pointer, wherein the other mempool node maintains a set of weight associated connections to a set of mempool nodes, wherein the weight is proportional to the level of trust in the connection.

The invention provides a system and a method for deploying a model based on a cluster. The resource utilization rate of the model after being deployed in the cluster can be improved. The system based on the cluster deployment model comprises a scheduling controller which is used for receiving a reasoning request, adjusting resource distribution of a multi-model shared container group and distributing the reasoning request to the multi-model shared container group to which a corresponding model belongs; the multi-model sharing container group is in communication connection with the dispatching controller, and the multi-model sharing container group is used for simultaneously operating a plurality of models of the same type; and the distributed memory is in communication connection with the multi-model sharing container group, and the distributed memory is used for storing model file data and model metadata of the multi-model sharing container group.