1405 results about "Object storage" patented technology

The invention discloses a geographic information big data-oriented multi-source data processing system, and relates to the technical field of data acquisition and sensors, and the system comprises a data acquisition module which accesses a remote sensing satellite, an unmanned aerial vehicle, an Internet of Things sensor and social media in real time through a multi-source heterogeneous interface, and carries out the adaptive analysis of a data format and metadata marking; the distributed storage module is used for performing partition storage on the geographic information data based on a space-time database and an object storage architecture, and establishing dynamic space-time index and version control; the data fusion module is used for realizing coordinate system conversion, time sequence calibration and semantic knowledge graph matching based on a multi-source data alignment method of dynamic weight distribution; and an intelligent analysis module and a security control module. According to the method, core pain points such as data splitting, low storage efficiency, extensive analysis and compliance risks in the geographic information field are systematically solved, and a full-stack type technical base is provided for scenes such as smart cities, emergency disaster relief and environment monitoring.

A database system integrates in-database machine learning (ML) models with in-database large language models (LLMs) or other generative artificial intelligence (AI) models that enable new applications. The database system receives one or more inferences from an ML model and provides an inference input to a retrieval agent of an object store. One or more vector stores represent a plurality of reference documents using semantic encodings. The retrieval agent performs a similarity search of the one or more vector stores to retrieve a set of passages from the plurality of reference documents based on similarity of encodings of the inference input and encodings of passages in the plurality of reference documents. The database system generates a linguistic prompt for an LLM having a context including the inferences and passages and applies the LLM to the linguistic prompt to generate a natural language explanation of the one or more inferences.

The invention discloses a file uploading method and device, a storage medium, electronic equipment and a program product, and relates to the technical field of computer networks and distributed storage, and the method comprises the steps: collecting metadata of a to-be-uploaded file and a network state of an uploading path between a client and a server in real time; calculating the size of a target fragment of the to-be-uploaded file for data uploading between the client and the server through the metadata and the network state; segmenting the to-be-uploaded file based on the target fragment size to obtain a plurality of data blocks; according to the method, the plurality of data blocks are used for uploading the file from the client to the server, and the uploading parameters of the different data blocks are recorded, so that the uploading result of the file to be uploaded transmitted to the server is determined through the uploading parameters, the technical problem that the uploading efficiency and robustness of the object storage system are poor is solved, and the uploading efficiency of the object storage system is improved. The technical effect of improving the uploading efficiency and robustness of the object storage system is achieved.

The invention relates to the technical field of geographic information systems, in particular to a map tile data efficient access method and system based on hybrid storage and intelligent layering, and the method comprises the following steps: constructing a multi-level storage architecture, carrying out dynamic heat analysis, carrying out intelligent layering scheduling, preloading and space prediction, and carrying out transmission and access optimization. The method has the beneficial effects that the multi-level storage pool is constructed by combining the advantages of object storage and a distributed file system; adopting a dynamic tile popularity analysis model to realize hot data cache acceleration and cold data hierarchical archiving; and meanwhile, a tile request preloading mechanism is introduced, and an access path is optimized in combination with a spatial locality prediction algorithm. According to the method, the storage cost can be remarkably reduced, the tile request delay is reduced, quick response in a high-concurrency scene is supported, and the method is suitable for application scenes such as Internet map services and smart cities.

The invention discloses a permanent incremental backup method based on an S3 bucket-level virtual snapshot, and relates to the technical field of data storage and backup, the method comprises the following steps: initializing a backup system and starting a backup agent program, an index management service and an S3 object storage service; complete backup is executed, and a VID and S3 bucket-level virtual snapshoot are generated; efficiently managing a mapping relation between a file path and a snapshot identifier through a B + tree and an inverted index by utilizing a MongoDB index service; incremental backup is executed, and file changes are accurately recognized through Hash comparison and modification time comparison; a storage position is optimized by adopting an incremental storage and deduplication storage mechanism and an IncDSI optimization algorithm; executing backup chain integration and cleaning; and performing data recovery based on the VID of the file and the directory. According to the method, the backup efficiency is remarkably improved, the storage overhead is reduced, the data recovery process is optimized, and the method is suitable for data backup and management of a large-scale file system and has important application value and prospect.

Embodiments are directed to a file system that include object stores. An object store for write requests may be provided. Write ahead log (WAL) entries that include data blocks may be generated. A WAL object may be generated based on the WAL entries and stored in the object store. An in-memory overlay may be updated to associate the data blocks with the WAL object. A checkpoint operation may be executed to: generate an index object that includes index entries that associate other data blocks with data objects stored in the object store; update the index object to include index entries that associate the data blocks with the WAL object; store the updated index object in the object store; update the in-memory overlay to remove the association of the data blocks and the WAL object and update the in memory WAL to remove records of successfully checkpointed WAL objects.

The invention relates to a method and a system for storing and backing up electric power data of a data-in-data station based on use popularity, and belongs to the technical field of data-in-data stations, and the method comprises the following steps: classifying electric power business data into hot data, temperature data and cold data based on preset access frequency, time window and data volume index; according to a classification result and a business importance degree, storing hot data in an SSD cloud disk or a memory database, storing temperature data in a distributed column database, and compressing cold data and then archiving the cold data to an object storage or a tape library; dynamically configuring differentiated backup strategies for different types of data; automatic scheduling of storage and backup is realized through a cloud platform component, and hot switching of master and slave databases, OSS archived data recovery by a DataWorks tool and a remote multi-copy recovery mechanism are supported; and migrating data in advance in combination with a historical access prediction popularity trend, and triggering cold data compression and archiving when storage resources exceed 80%. The method is suitable for power industry scenes, the storage cost is optimized, and the data calling efficiency is improved.

Storing and retrieving data using vector embeddings, including: receiving a request to store a data object in an object store; generating, based on the data object, a vector embedding for the data object; and storing, in the object store, the data object and the vector embedding for the data object.

The invention discloses a garbage collection method and device, equipment and a storage medium, and relates to the technical field of electrical digital data processing.The method comprises the steps that when metadata of a cache pool is updated, internal object abstracts corresponding to the metadata are put into a garbage collection queue; according to the water level condition of the data pool and the disk utilization rate of the object storage device of the data pool, the single-round execution task number of the garbage collection queue is determined; according to the number of single-round execution tasks of the garbage collection queue and the garbage collection priorities of the internal objects, the internal objects are selected to execute garbage collection, so that effective data in the internal objects are aggregated into new internal objects, and the internal objects subjected to garbage collection are deleted, namely, the situation that the water level of the current data pool is too high, and the garbage collection efficiency is improved is avoided. And the overlarge disk utilization rate of the object storage equipment is avoided, and the two are balanced, so that the influence on other businesses is avoided as far as possible on the basis of ensuring the normal recovery of the garbage.

The invention discloses a data storage method for an artificial intelligence learning mode, and relates to the technical field of computer data storage, and the method comprises the steps: 1, merging a multi-source perception stream into blocks in real time at the edge through monotone serial number writing, so as to provide a replayable time sequence; 2, asynchronous erasure coding is executed on the blocks, Merkel roots are calculated and written into a local cache, and dual guarantee of loss tolerance and integrity is achieved; 3, pushing slices and roots to object storage in sequence according to a network, and calling a time travel interface to solidify an incremental snapshot; 4, the cloud end monitors a snapshot hash event, a serial number chain is written through differential scanning, a gap is reconstructed through slices, an index is refreshed, and continuous consistency is kept; 5, the training process generates a Merkel proof online verification sample, and damaged data are immediately interpolated and repaired and an audit chain is recorded; and step 6, after training is finished, generating a leatherwise list and a frozen root, asynchronously cleaning redundant slices, updating a version table, and finally forming single-fingerprint traceable cost archiving.

The invention discloses a method and system for generating a multi-modal report based on a large language model and a FreeMarker, and belongs to the technical field of natural language processing, and the method comprises the following steps: step 1, professional data preprocessing; 2, splicing the preprocessed data and prompt, and performing data analysis and text generation by using a large language model; 3, analyzing a model output result to obtain a corresponding analysis conclusion, data statistics and a reference document, and constructing a plurality of representation forms of the content by means of object storage; step 4, designing a report template in the FreeMarker; and step 5, filling the content generated by the large language model into the FreeMarker template. According to the method, the problems of single report generation form and poor display form and information hierarchy of a traditional large language model can be solved, so that the professional field recognition degree of the generated report is improved, and the requirements of various industries for high-quality and diversified reports are met.

The invention discloses a solid state disk management method and device, equipment, a medium and a product, which are applied to the technical field of storage and are used for solving the problem of data loss caused by difficult realization of timely and accurate solid state disk anomaly detection in the traditional technology, and the method is applied to a distributed storage system deployed with a service layer and a storage layer. The storage layer is deployed with a plurality of object storage devices, each object storage device is deployed with a plurality of solid state disks, and the method comprises the following steps: monitoring a service layer to determine whether the service layer is abnormal or not; if yes, determining an abnormal solid state disk and a solid state disk exception type in a storage layer; the exception type of the solid-state disk comprises exception of the disk and / or expiration of the service life of the disk; when the exception type of the solid state disk is the exception of the solid state disk, performing fault detection on the exceptional solid state disk by utilizing a preset fault detection strategy to obtain a fault detection result; and determining a target protection strategy by using the solid state disk exception type and the fault detection result so as to perform redundancy protection on the exceptional solid state disk.

Described is a system for cloud-based processing of backup data for storage onto various types of object storage systems. A specialized cloud-based component (e.g. proxy server) may act as an intermediary when backing up data from a client system to an object storage. For example, the cloud-based component may be provided as part of backup service that operates in conjunction with a client system. Accordingly, the system may allow different clients of a backup service to efficiently and seamlessly interact with the different third-party object storage providers via a single cloud-based component. To provide such object-storage-agnostic functionality, the cloud-based component may include a function library that includes object-storage-specific functions. For example, the cloud-based component may invoke a specific set of operations that interact with a storage API provided by a particular object storage provider.

The embodiment of the invention relates to a distributed object storage method and device, equipment and a medium. The method comprises the steps that a to-be-stored target object is split into multiple pieces of key value pair data; determining target partitions corresponding to the plurality of key value pair data from a plurality of partitions of a preset distributed object storage system; determining a data storage node currently holding the target partition based on a preset partition view to obtain data storage nodes corresponding to the plurality of key value pair data; wherein the partition view is used for indicating meta-information of the plurality of partitions, and the meta-information at least comprises information of data storage nodes holding the partitions; the data storage nodes are storage nodes of the distributed object storage system; and caching the plurality of key value pair data to respective corresponding data storage nodes. According to the embodiment of the invention, the cluster local overload problem of the object storage service can be effectively relieved.

A heterogeneous storage-oriented hot and cold data management method and system, wherein hot and cold data are managed by means of the following steps: at table granularity, storing data from each table into an independent data bucket in the form of object storage, each data bucket comprising a plurality of partitions, the data being randomly distributed in each partition, and the consistency of replicas of the partitioned data being ensured by means of a distributed consensus algorithm; collecting statistics about the access frequencies of the partitions in one data bucket; when the access frequency of one partition reaches a set threshold, adding a new replica to a target storage medium by means of a membership change in the distributed consensus algorithm; and when data synchronization between the new replica and an old replica is completed, detecting the old replica. In this way, the migration of the partitioned data on different storage media is realized, and data cooling and heating are implemented. The method and system can maximize the use efficiency of high-performance storage media, ensure the access efficiency of users, and can reduce the costs of data migration and maintenance.

The invention particularly relates to an object storage data encryption method based on a large model. According to the object storage data encryption method based on the large model, user data is partitioned and compressed, and a deep learning large model is trained by using historical data accumulated by a cloud storage provider and known security threats; according to a large model prediction result, selecting an optimal encryption algorithm and a key management strategy; encrypting the preprocessed data, and storing the encrypted data and metadata in a cloud storage system; and when the user needs to access the data, reading the metadata, obtaining the decryption key, and recovering the original data. The object storage data encryption method based on the large model is efficient, safe and flexible, the optimal encryption strategy can be intelligently selected according to changes of data characteristics and the safety environment, dynamic encryption is achieved, therefore, the safety and privacy of data are remarkably improved, and the requirements of a modern object storage system can be met.

The invention provides an abnormity early warning security system and method based on cross-modal semantic retrieval, and relates to the technical field of public security, and the system comprises a video collection and frame sampling module which is used for extracting key frame images from a real-time monitoring video stream according to a preset frame interval and storing the key frame images to an object storage system; the target detection module adopts a YOLO model to execute instance-level target detection on the extraction frame, and generates a structured detection result containing a target position, a category and confidence; the semantic vector storage and structured warehousing module packages the standardized metadata of the image frame and the semantic vector generated by the CLIP into structured data, and stores the structured data into a high-performance vector database; and the semantic retrieval and reverse query module receives a natural language query instruction to generate a semantic vector, and realizes cross-modal retrieval of historical monitoring images through vector similarity matching. According to the invention, a set of exception security system with real-time perception capability, semantic understanding capability and cross-modal retrieval capability is constructed.

Embodiments are directed to a file system that include object stores. An object store for write requests may be provided. Write ahead log (WAL) entries that include data blocks may be generated. A WAL object may be generated based on the WAL entries and stored in the object store. An in-memory overlay may be updated to associate the data blocks with the WAL object. A checkpoint operation may be executed to: generate an index object that includes index entries that associate other data blocks with data objects stored in the object store; update the index object to include index entries that associate the data blocks with the WAL object; store the updated index object in the object store; update the in-memory overlay to remove the association of the data blocks and the WAL object and update the in memory WAL to remove records of successfully checkpointed WAL objects.

An elector is provided using shared storage to allow a cluster of participant nodes to gracefully convert failed voters to non-voters until there are only two voters. The cluster can then tolerate the failure of either of those two remaining participant nodes. A tiny amount of shared storage is used as a voter without an object store. This results in a cluster with only one running node that is an exclusive leader. An exclusive leader is then the only participant node that is a voter, and the exclusive leader can commit log records without replicating them to any other members of the cluster. When there is an exclusive leader, there is only one remaining node with a current copy of the object store.

Devices, networks, systems, methods, and processes for utilizing a time-based queue are described herein. A device may receive an object stream comprising multiple objects. The device may assign a Time-To-Live (TTL) value for each object in the object stream and a maximum TTL value for the object stream. The device may create and store the time-based queue in a memory based on the maximum TTL value. The device may insert the objects in the time-based queue at corresponding object insertion times. The device can pop the objects out of the time-based queue in First-In-First-Out (FIFO) order. The objects may expire when time periods indicated by corresponding TTL values elapse after the corresponding object insertion times. The device can further remove the expired objects from the time-based queue. The time-based queue stored in the memory can include references to the objects stored in a time-based storage in form of an array.

A distributed task scheduling system based on RabbitMQ, Redis and MongoDB. The task scheduling system comprises a task system service, Redis, MongoDB, RabbitMQ and an SDK, wherein the task system service comprises a data and object storage management module, a message queue management module, a task management module, a timing task module and a workflow module. By means of the design of a task system service, Redis, MongoDB, RabbitMQ and an SDK, the functions of an asynchronous task, a timing task and a workflow in a distributed and micro-service environment are provided. By means of the technical solution of a message queue, trigger-based scheduling of a task is realized, and a reliable and unique task scheduling capability exists; moreover, a timing task trigger capability exists, and thus the present invention is suitable for a scenario in which the execution of a certain task is triggered in a timing or delayed manner; a workflow capability is provided, and thus serial and parallel execution of a plurality of tasks can be performed; and capabilities such as task rollback and interruption are provided, and thus a system which unifies the asynchronous task, the timing task and the workflow is achieved.

The invention discloses a cloud storage service implementation method and system for unified management of multiple types of storage, and belongs to the technical field of cloud storage, and the method specifically comprises the steps: building a unified abstract model oriented to heterogeneous storage media, mapping distributed files, object storage and block storage devices into standardized storage units, deploying a cloud block chain, and carrying out uplink, a storage index table under a chain is constructed, an index state is dynamically updated through a loose coupling consensus mechanism, the loose coupling consensus mechanism comprises a weighted voting algorithm based on a storage node reputation value and an asynchronous verification mechanism based on space-time certification, and data is divided into data pieces and verification pieces; fragmentation information is written into a storage metadata registration chain and a service quality proof chain at the same time, cross-chain storage is carried out on data, and cloud storage resources are dynamically distributed; according to the method, the storage management function is decoupled through the block chain, so that the retrieval accuracy is improved, the retrieval response time is effectively shortened, and the data transmission cost is reduced.

The invention provides a data system, a data management method, a data management device, a data query method and a data query device.The data system comprises an object storage layer, at least one ordered character string table SSTable is stored in the object storage layer, a plurality of data blocks and an index tree are stored in the SSTable, data and index information of the data are stored in the data blocks, and the index tree is connected with the object storage layer; the index tree comprises a plurality of nodes of a tree structure, leaf nodes in the nodes correspond to the data blocks one to one, and index information in the corresponding data blocks is stored in the leaf nodes; and the at least one database is in communication connection with the object storage layer and is used for externally providing data service based on the data stored in the object storage layer.

The invention discloses a dynamic expansion method and system for an S3 standard object storage interface and a medium, and belongs to the technical field of computer storage. The method comprises the following steps: a system initialization stage; in the uploading request stage, the file uploading request is subjected to dual verification, and if verification is passed, the file uploading request is sent to the file processing module; in the request processing stage, the file processing module verifies the received file uploading request again and executes different processing flows according to the type of the file uploading request; in the flow control recording stage, flow control and flow recording are carried out according to the uploaded flow data; and in the exception and security event processing stage, in the file uploading process, if any module has an internal error, an error report is immediately generated, and exception processing is carried out. Through triple verification, the data security is ensured, different processing modes are made according to different request types, and the processing efficiency is improved.

The invention provides a data transmission method, device and equipment and a readable storage medium, and the method comprises the steps: obtaining the file attribute information of a to-be-transmitted file, and enabling fragmentation transmission if the file attribute information meets a preset condition; hardware resource information is obtained, configuration information of fragment transmission is configured according to the hardware resource information, and the configuration information at least comprises the fragment size; executing the fragmentation transmission according to the configuration information of the fragmentation transmission; the file to be transmitted is a file to be sent by the opposite terminal equipment and / or a file to be acquired by the opposite terminal equipment, and the fragment transmission comprises fragment receiving of the file from the opposite terminal equipment and / or fragment sending of the file to the opposite terminal equipment. Through the technical scheme of the specification, the size of the fragment is dynamically adjusted, the transmission efficiency of the large file is optimized, the memory pressure is reduced, the network retry cost is reduced, and the object storage service performance is improved.

The invention relates to the technical field of data storage, and provides an object storage intelligent layering method and device based on a decision tree, and the method comprises the steps: collecting object data, carrying out the preprocessing of the collected object data, and setting an access time window and a storage label for the obtained standardized object data; performing feature extraction on the stored standardized data, and performing dimension reduction processing on the extracted features to obtain feature data; constructing a training model, training and testing the training model by adopting feature data, and selecting a decision tree model from the training model for production deployment; and performing classified storage prediction on the uploaded object data by adopting the deployed decision tree model, and storing the object data to a corresponding storage pool according to a classified storage prediction result. According to the method, the future access frequency of the object can be predicted and directly allocated to different storage medium layers, so that the data migration time and space loss are saved, the storage system efficiency is improved, and the user access performance is improved.

The invention belongs to the field of big data, and relates to a file storage method which comprises the steps that access frequency information of a target file in a historical time period is acquired, and the target file is stored in a mechanical hard disk of an object storage system; judging whether the target file is a high-frequency access file or not according to the access frequency information and an access threshold value; if the target file is the high-frequency access file, obtaining file attribute information of the target file, and judging a to-be-stored type of the target file based on the cache strategy and the file attribute information; if the to-be-stored type of the target file is a memory storage type, storing the target file into a memory cache; and if the to-be-stored type of the target file is the standard storage type, storing the target file into a solid state disk cache. The invention further provides a device, equipment and a medium. In addition, the invention also relates to a block chain technology, and the access frequency information and the target file can be stored in a block chain. The method can improve the utilization rate of storage resources.

The invention provides a component configuration method which can be applied to the technical field of cloud computing and the technical field of big data. The method comprises the steps of obtaining configuration information used for configuring a target interface component from a server side based on an interface request; packaging the interface request into an asynchronous data acquisition object, and storing the asynchronous data acquisition object as a global shared instance; registering a callback processing function based on the global shared instance for the target interface component; and responding to the configuration information returned by the server side and triggering the callback processing function, and executing component rendering on the target interface component based on the configuration information.

Embodiments are directed to a file system that include object stores. An object store for write requests may be provided. Write ahead log (WAL) entries that include data blocks may be generated. A WAL object may be generated based on the WAL entries and stored in the object store. An in-memory overlay may be updated to associate the data blocks with the WAL object. A checkpoint operation may be executed to: generate an index object that includes index entries that associate other data blocks with data objects stored in the object store; update the index object to include index entries that associate the data blocks with the WAL object; store the updated index object in the object store; update the in-memory overlay to remove the association of the data blocks and the WAL object and update the in memory WAL to remove records of successfully checkpointed WAL objects.

The invention relates to the technical field of research and development management, can be applied to business scenes of financial science and technology, medical health and the like, and discloses an anomaly analysis method, device, equipment and medium based on a variation test. Generating a test case through the semantic perception model; selecting a variation strategy by using a state-driven model according to the service state, and generating a variation test case based on the access control strategy; executing the variation test case in a distributed environment, and collecting system resource index data and request execution monitoring data; and inputting the collected data into the abnormal prediction model, and outputting an abnormal behavior prediction result. Through combination of semantic perception generation, state-driven selection and anomaly prediction, coverage of a test case on protocol dependence and a service state is improved, and the accuracy of anomaly detection of the object storage system is remarkably improved.