1075 results about "Object storage" patented technology

The invention discloses a geographic information big data-oriented multi-source data processing system, and relates to the technical field of data acquisition and sensors, and the system comprises a data acquisition module which accesses a remote sensing satellite, an unmanned aerial vehicle, an Internet of Things sensor and social media in real time through a multi-source heterogeneous interface, and carries out the adaptive analysis of a data format and metadata marking; the distributed storage module is used for performing partition storage on the geographic information data based on a space-time database and an object storage architecture, and establishing dynamic space-time index and version control; the data fusion module is used for realizing coordinate system conversion, time sequence calibration and semantic knowledge graph matching based on a multi-source data alignment method of dynamic weight distribution; and an intelligent analysis module and a security control module. According to the method, core pain points such as data splitting, low storage efficiency, extensive analysis and compliance risks in the geographic information field are systematically solved, and a full-stack type technical base is provided for scenes such as smart cities, emergency disaster relief and environment monitoring.

A database system integrates in-database machine learning (ML) models with in-database large language models (LLMs) or other generative artificial intelligence (AI) models that enable new applications. The database system receives one or more inferences from an ML model and provides an inference input to a retrieval agent of an object store. One or more vector stores represent a plurality of reference documents using semantic encodings. The retrieval agent performs a similarity search of the one or more vector stores to retrieve a set of passages from the plurality of reference documents based on similarity of encodings of the inference input and encodings of passages in the plurality of reference documents. The database system generates a linguistic prompt for an LLM having a context including the inferences and passages and applies the LLM to the linguistic prompt to generate a natural language explanation of the one or more inferences.

The invention discloses a file uploading method and device, a storage medium, electronic equipment and a program product, and relates to the technical field of computer networks and distributed storage, and the method comprises the steps: collecting metadata of a to-be-uploaded file and a network state of an uploading path between a client and a server in real time; calculating the size of a target fragment of the to-be-uploaded file for data uploading between the client and the server through the metadata and the network state; segmenting the to-be-uploaded file based on the target fragment size to obtain a plurality of data blocks; according to the method, the plurality of data blocks are used for uploading the file from the client to the server, and the uploading parameters of the different data blocks are recorded, so that the uploading result of the file to be uploaded transmitted to the server is determined through the uploading parameters, the technical problem that the uploading efficiency and robustness of the object storage system are poor is solved, and the uploading efficiency of the object storage system is improved. The technical effect of improving the uploading efficiency and robustness of the object storage system is achieved.

The invention relates to the technical field of geographic information systems, in particular to a map tile data efficient access method and system based on hybrid storage and intelligent layering, and the method comprises the following steps: constructing a multi-level storage architecture, carrying out dynamic heat analysis, carrying out intelligent layering scheduling, preloading and space prediction, and carrying out transmission and access optimization. The method has the beneficial effects that the multi-level storage pool is constructed by combining the advantages of object storage and a distributed file system; adopting a dynamic tile popularity analysis model to realize hot data cache acceleration and cold data hierarchical archiving; and meanwhile, a tile request preloading mechanism is introduced, and an access path is optimized in combination with a spatial locality prediction algorithm. According to the method, the storage cost can be remarkably reduced, the tile request delay is reduced, quick response in a high-concurrency scene is supported, and the method is suitable for application scenes such as Internet map services and smart cities.

The invention discloses a permanent incremental backup method based on an S3 bucket-level virtual snapshot, and relates to the technical field of data storage and backup, the method comprises the following steps: initializing a backup system and starting a backup agent program, an index management service and an S3 object storage service; complete backup is executed, and a VID and S3 bucket-level virtual snapshoot are generated; efficiently managing a mapping relation between a file path and a snapshot identifier through a B + tree and an inverted index by utilizing a MongoDB index service; incremental backup is executed, and file changes are accurately recognized through Hash comparison and modification time comparison; a storage position is optimized by adopting an incremental storage and deduplication storage mechanism and an IncDSI optimization algorithm; executing backup chain integration and cleaning; and performing data recovery based on the VID of the file and the directory. According to the method, the backup efficiency is remarkably improved, the storage overhead is reduced, the data recovery process is optimized, and the method is suitable for data backup and management of a large-scale file system and has important application value and prospect.

Embodiments are directed to a file system that include object stores. An object store for write requests may be provided. Write ahead log (WAL) entries that include data blocks may be generated. A WAL object may be generated based on the WAL entries and stored in the object store. An in-memory overlay may be updated to associate the data blocks with the WAL object. A checkpoint operation may be executed to: generate an index object that includes index entries that associate other data blocks with data objects stored in the object store; update the index object to include index entries that associate the data blocks with the WAL object; store the updated index object in the object store; update the in-memory overlay to remove the association of the data blocks and the WAL object and update the in memory WAL to remove records of successfully checkpointed WAL objects.

The invention relates to a method and a system for storing and backing up electric power data of a data-in-data station based on use popularity, and belongs to the technical field of data-in-data stations, and the method comprises the following steps: classifying electric power business data into hot data, temperature data and cold data based on preset access frequency, time window and data volume index; according to a classification result and a business importance degree, storing hot data in an SSD cloud disk or a memory database, storing temperature data in a distributed column database, and compressing cold data and then archiving the cold data to an object storage or a tape library; dynamically configuring differentiated backup strategies for different types of data; automatic scheduling of storage and backup is realized through a cloud platform component, and hot switching of master and slave databases, OSS archived data recovery by a DataWorks tool and a remote multi-copy recovery mechanism are supported; and migrating data in advance in combination with a historical access prediction popularity trend, and triggering cold data compression and archiving when storage resources exceed 80%. The method is suitable for power industry scenes, the storage cost is optimized, and the data calling efficiency is improved.

The invention discloses a garbage collection method and device, equipment and a storage medium, and relates to the technical field of electrical digital data processing.The method comprises the steps that when metadata of a cache pool is updated, internal object abstracts corresponding to the metadata are put into a garbage collection queue; according to the water level condition of the data pool and the disk utilization rate of the object storage device of the data pool, the single-round execution task number of the garbage collection queue is determined; according to the number of single-round execution tasks of the garbage collection queue and the garbage collection priorities of the internal objects, the internal objects are selected to execute garbage collection, so that effective data in the internal objects are aggregated into new internal objects, and the internal objects subjected to garbage collection are deleted, namely, the situation that the water level of the current data pool is too high, and the garbage collection efficiency is improved is avoided. And the overlarge disk utilization rate of the object storage equipment is avoided, and the two are balanced, so that the influence on other businesses is avoided as far as possible on the basis of ensuring the normal recovery of the garbage.

The invention discloses a data storage method for an artificial intelligence learning mode, and relates to the technical field of computer data storage, and the method comprises the steps: 1, merging a multi-source perception stream into blocks in real time at the edge through monotone serial number writing, so as to provide a replayable time sequence; 2, asynchronous erasure coding is executed on the blocks, Merkel roots are calculated and written into a local cache, and dual guarantee of loss tolerance and integrity is achieved; 3, pushing slices and roots to object storage in sequence according to a network, and calling a time travel interface to solidify an incremental snapshot; 4, the cloud end monitors a snapshot hash event, a serial number chain is written through differential scanning, a gap is reconstructed through slices, an index is refreshed, and continuous consistency is kept; 5, the training process generates a Merkel proof online verification sample, and damaged data are immediately interpolated and repaired and an audit chain is recorded; and step 6, after training is finished, generating a leatherwise list and a frozen root, asynchronously cleaning redundant slices, updating a version table, and finally forming single-fingerprint traceable cost archiving.

A heterogeneous storage-oriented hot and cold data management method and system, wherein hot and cold data are managed by means of the following steps: at table granularity, storing data from each table into an independent data bucket in the form of object storage, each data bucket comprising a plurality of partitions, the data being randomly distributed in each partition, and the consistency of replicas of the partitioned data being ensured by means of a distributed consensus algorithm; collecting statistics about the access frequencies of the partitions in one data bucket; when the access frequency of one partition reaches a set threshold, adding a new replica to a target storage medium by means of a membership change in the distributed consensus algorithm; and when data synchronization between the new replica and an old replica is completed, detecting the old replica. In this way, the migration of the partitioned data on different storage media is realized, and data cooling and heating are implemented. The method and system can maximize the use efficiency of high-performance storage media, ensure the access efficiency of users, and can reduce the costs of data migration and maintenance.

The invention particularly relates to an object storage data encryption method based on a large model. According to the object storage data encryption method based on the large model, user data is partitioned and compressed, and a deep learning large model is trained by using historical data accumulated by a cloud storage provider and known security threats; according to a large model prediction result, selecting an optimal encryption algorithm and a key management strategy; encrypting the preprocessed data, and storing the encrypted data and metadata in a cloud storage system; and when the user needs to access the data, reading the metadata, obtaining the decryption key, and recovering the original data. The object storage data encryption method based on the large model is efficient, safe and flexible, the optimal encryption strategy can be intelligently selected according to changes of data characteristics and the safety environment, dynamic encryption is achieved, therefore, the safety and privacy of data are remarkably improved, and the requirements of a modern object storage system can be met.

The invention provides an abnormity early warning security system and method based on cross-modal semantic retrieval, and relates to the technical field of public security, and the system comprises a video collection and frame sampling module which is used for extracting key frame images from a real-time monitoring video stream according to a preset frame interval and storing the key frame images to an object storage system; the target detection module adopts a YOLO model to execute instance-level target detection on the extraction frame, and generates a structured detection result containing a target position, a category and confidence; the semantic vector storage and structured warehousing module packages the standardized metadata of the image frame and the semantic vector generated by the CLIP into structured data, and stores the structured data into a high-performance vector database; and the semantic retrieval and reverse query module receives a natural language query instruction to generate a semantic vector, and realizes cross-modal retrieval of historical monitoring images through vector similarity matching. According to the invention, a set of exception security system with real-time perception capability, semantic understanding capability and cross-modal retrieval capability is constructed.

Embodiments are directed to a file system that include object stores. An object store for write requests may be provided. Write ahead log (WAL) entries that include data blocks may be generated. A WAL object may be generated based on the WAL entries and stored in the object store. An in-memory overlay may be updated to associate the data blocks with the WAL object. A checkpoint operation may be executed to: generate an index object that includes index entries that associate other data blocks with data objects stored in the object store; update the index object to include index entries that associate the data blocks with the WAL object; store the updated index object in the object store; update the in-memory overlay to remove the association of the data blocks and the WAL object and update the in memory WAL to remove records of successfully checkpointed WAL objects.

A distributed task scheduling system based on RabbitMQ, Redis and MongoDB. The task scheduling system comprises a task system service, Redis, MongoDB, RabbitMQ and an SDK, wherein the task system service comprises a data and object storage management module, a message queue management module, a task management module, a timing task module and a workflow module. By means of the design of a task system service, Redis, MongoDB, RabbitMQ and an SDK, the functions of an asynchronous task, a timing task and a workflow in a distributed and micro-service environment are provided. By means of the technical solution of a message queue, trigger-based scheduling of a task is realized, and a reliable and unique task scheduling capability exists; moreover, a timing task trigger capability exists, and thus the present invention is suitable for a scenario in which the execution of a certain task is triggered in a timing or delayed manner; a workflow capability is provided, and thus serial and parallel execution of a plurality of tasks can be performed; and capabilities such as task rollback and interruption are provided, and thus a system which unifies the asynchronous task, the timing task and the workflow is achieved.

The invention discloses a cloud storage service implementation method and system for unified management of multiple types of storage, and belongs to the technical field of cloud storage, and the method specifically comprises the steps: building a unified abstract model oriented to heterogeneous storage media, mapping distributed files, object storage and block storage devices into standardized storage units, deploying a cloud block chain, and carrying out uplink, a storage index table under a chain is constructed, an index state is dynamically updated through a loose coupling consensus mechanism, the loose coupling consensus mechanism comprises a weighted voting algorithm based on a storage node reputation value and an asynchronous verification mechanism based on space-time certification, and data is divided into data pieces and verification pieces; fragmentation information is written into a storage metadata registration chain and a service quality proof chain at the same time, cross-chain storage is carried out on data, and cloud storage resources are dynamically distributed; according to the method, the storage management function is decoupled through the block chain, so that the retrieval accuracy is improved, the retrieval response time is effectively shortened, and the data transmission cost is reduced.

The invention provides a data transmission method, device and equipment and a readable storage medium, and the method comprises the steps: obtaining the file attribute information of a to-be-transmitted file, and enabling fragmentation transmission if the file attribute information meets a preset condition; hardware resource information is obtained, configuration information of fragment transmission is configured according to the hardware resource information, and the configuration information at least comprises the fragment size; executing the fragmentation transmission according to the configuration information of the fragmentation transmission; the file to be transmitted is a file to be sent by the opposite terminal equipment and / or a file to be acquired by the opposite terminal equipment, and the fragment transmission comprises fragment receiving of the file from the opposite terminal equipment and / or fragment sending of the file to the opposite terminal equipment. Through the technical scheme of the specification, the size of the fragment is dynamically adjusted, the transmission efficiency of the large file is optimized, the memory pressure is reduced, the network retry cost is reduced, and the object storage service performance is improved.

The invention relates to the technical field of data storage, and provides an object storage intelligent layering method and device based on a decision tree, and the method comprises the steps: collecting object data, carrying out the preprocessing of the collected object data, and setting an access time window and a storage label for the obtained standardized object data; performing feature extraction on the stored standardized data, and performing dimension reduction processing on the extracted features to obtain feature data; constructing a training model, training and testing the training model by adopting feature data, and selecting a decision tree model from the training model for production deployment; and performing classified storage prediction on the uploaded object data by adopting the deployed decision tree model, and storing the object data to a corresponding storage pool according to a classified storage prediction result. According to the method, the future access frequency of the object can be predicted and directly allocated to different storage medium layers, so that the data migration time and space loss are saved, the storage system efficiency is improved, and the user access performance is improved.

The invention provides a component configuration method which can be applied to the technical field of cloud computing and the technical field of big data. The method comprises the steps of obtaining configuration information used for configuring a target interface component from a server side based on an interface request; packaging the interface request into an asynchronous data acquisition object, and storing the asynchronous data acquisition object as a global shared instance; registering a callback processing function based on the global shared instance for the target interface component; and responding to the configuration information returned by the server side and triggering the callback processing function, and executing component rendering on the target interface component based on the configuration information.

Embodiments are directed to a file system that include object stores. An object store for write requests may be provided. Write ahead log (WAL) entries that include data blocks may be generated. A WAL object may be generated based on the WAL entries and stored in the object store. An in-memory overlay may be updated to associate the data blocks with the WAL object. A checkpoint operation may be executed to: generate an index object that includes index entries that associate other data blocks with data objects stored in the object store; update the index object to include index entries that associate the data blocks with the WAL object; store the updated index object in the object store; update the in-memory overlay to remove the association of the data blocks and the WAL object and update the in memory WAL to remove records of successfully checkpointed WAL objects.

The invention relates to the technical field of research and development management, can be applied to business scenes of financial science and technology, medical health and the like, and discloses an anomaly analysis method, device, equipment and medium based on a variation test. Generating a test case through the semantic perception model; selecting a variation strategy by using a state-driven model according to the service state, and generating a variation test case based on the access control strategy; executing the variation test case in a distributed environment, and collecting system resource index data and request execution monitoring data; and inputting the collected data into the abnormal prediction model, and outputting an abnormal behavior prediction result. Through combination of semantic perception generation, state-driven selection and anomaly prediction, coverage of a test case on protocol dependence and a service state is improved, and the accuracy of anomaly detection of the object storage system is remarkably improved.

In one embodiment, a computer implemented method includes storing objects in a first bucket and storing files in a second bucket of a first storage cluster of the distributed storage system, synchronously replicating data of the objects into a third mirrored bucket of a second storage cluster of the distributed storage system, synchronously replicating OSP configuration data from the first storage cluster to the second storage cluster during the synchronous replication, and providing non-disruptive operations with zero recovery time objective (RTO), and ensuring consistency between the objects in the first bucket and the objects in the third bucket for a software application that is accessing one or more objects and files using the OSP. The objects and files are accessible through an object storage protocol (OSP).

Systems and methods for performing fingerprint-based data loss prevention (DLP) using information obtained from a cloud-native service are provided. In one example, DLP fingerprints are obtained by a security enforcement system (e.g., a network security appliance, a secure access service edge (SASE) platform, or a security service edge (SSE)). The DLP fingerprints may be generated locally by the security enforcement system or generated remotely from the security service edge based on a set of files stored in a cloud-native service, representing, for example, infrastructure-as-a-service (IaaS) (e.g., an object storage service) or Software-as-a-service (SaaS) (e.g., a file hosting service or a productivity platform). Based at least in part on the DLP fingerprints, DLP is performed on a file by a DLP service of the security enforcement system, in which the file is at rest on an endpoint protected by the security enforcement system or in transit through the security enforcement point.

Examples described herein provide a computer-implemented method for storing data pages of a database in a remote storage. The method includes receiving a write request associated with one or more data pages of the database. The method further includes storing, in a first storage layer having a write ahead log, the write request associated with the one or more data pages. The method further includes storing, in a first write buffer, the received write request associated with the one or more data pages that each include a legacy data page identifier. The method further includes storing, in a second storage layer having a second write buffer, a sorted string table (SST) of the received write request. The method further includes storing, in a top level of a log structured merge (LSM) tree disposed in the remote storage, the SST from the second write buffer.

Systems and methods for novel uses and / or improvements to data migration. In particular, systems and methods for data migration of metadata stored in object storage from one container to another, especially in instances when the metadata is destined for ingestion by an artificial intelligence application. The systems and methods ensure that all metadata (e.g., metadata stored in object storage) is preserved during data migration, including metadata such as content type, object lock mode, object lock retain until date, and / or custom metadata.

The invention relates to the technical field of data processing, and discloses a data storage optimization method and system for a solid state disk. The method comprises the steps of performing pattern recognition on small object metadata through multi-dimensional access feature extraction, performing five-level popularity level division based on a flash memory perception popularity evaluation algorithm, performing multi-level storage region division on an SSD, performing intelligent resource allocation according to popularity levels and storage regions, and realizing dynamic adjustment processing through adaptive load monitoring. The technical problem of how to carry out intelligent hot-cold separation and dynamic hierarchical storage on the small object metadata to reduce the SSD storage cost in a massive small object storage environment generated by a cloud native application is solved.

This application relates to a text object indexing method, an object storage system and related devices. Among them, the method includes: after the front-end service module writes a text object into the key-value storage database system, it adds a write success message of the text object to the text write message queue; the text analysis scheduling module consumes the write success message in the text write message queue, generates a text analysis task for the text object and schedules the text analysis module to execute the text analysis task, obtains a set of representative keywords of the text object as the metadata of the text object and updates it to the key-value storage database system, and adds a metadata update message of the text object to the metadata update message queue; so that the index service module reads the metadata of the text object from the key-value storage database system according to the metadata update message of the text object, constructs an inverted index of the text object, and provides an index service. Through the present invention, complex condition retrieval of text objects in an object storage system is realized.

A data storage system is operable to store a plurality of records of a plurality of tables via a plurality of files in memory resources of an object storage system of a data storage system and to store configuration data indicating table metadata, mapping storage of the plurality of records of the plurality of tables via the plurality of files, via a metadata processing system of the data storage system. A record identification pipeline is generated for execution based on filtering parameter data and the configuration data. A filtered row set identifying a proper subset of the plurality of records meeting the filtering parameter data is generated based on executing the record identification by accessing at least one file of the plurality of files.

The invention provides a life cycle deletion optimization method for a distributed object storage system, which comprises the following steps of: S1, detecting the business volume of a cluster, starting a pre-scanning operation when the business volume is lower than a preset threshold value, and identifying an object to be deleted according with a life cycle rule; s2, respectively writing the to-be-deleted object information of different dates into corresponding omaps, and storing omap data in a disk according to a preset packaging frequency; s3, when the life cycle expires, directly reading the omap of the corresponding date, skipping a real-time scanning step, and executing batch deletion operation; and S4, recording statistical information such as the deletion quantity scanned every day on the xattr of the barrel, so that management software performs data trend analysis and operation and maintenance decision. According to the embodiment of the invention, the life cycle deletion efficiency can be improved, the metadata operation pressure is reduced, and the cluster storage space utilization rate is optimized.

The invention relates to the technical field of big data storage, and discloses a multi-level data storage and mixed query method and system, a storage medium and equipment, and the method comprises the steps: collecting original buried point data, carrying out real-time stream processing, storing the processed real-time data in a hot storage layer, carrying out timing offline batch processing on historical buried point data, and storing the processed real-time data in a hot storage layer; converting into an optimized storage format and storing in an object storage system; an extended metadata management system is constructed, and structure information, partition information and access frequency information of data in the object storage system are automatically extracted and managed; executing query, and reducing the data scanning amount through a query optimization strategy; automatically migrating data among different storage layers through a dynamic cold and hot data layering mechanism; a unified query interface is provided, a query request is distributed to a corresponding real-time processing system or an offline query system based on an intelligent routing strategy, and a query result is cached and returned, so that high-performance and low-cost storage and second-level query of data are realized.

A method for providing access to objects associated with a particular client in a cloud storage system is disclosed. The method includes the steps of establishing a connection with a user, providing a client namespace associated with the client to the user, where the client namespace represents objects stored on the cloud storage system and objects stored on a private storage system apart from the cloud storage system, receiving a request from the user to access an object stored on the private storage system, and providing information to the user to facilitate access to the object stored on the private storage system by said user. Other systems and methods are also disclosed. Important advantages of the present invention are facilitated by separating the logic for user access (control plane) from the actual storage (Storage plane). Private file system access can still be managed from the cloud, while keeping the client data private.