815 results about "Object storage" patented technology

A database system integrates in-database machine learning (ML) models with in-database large language models (LLMs) or other generative artificial intelligence (AI) models that enable new applications. The database system receives one or more inferences from an ML model and provides an inference input to a retrieval agent of an object store. One or more vector stores represent a plurality of reference documents using semantic encodings. The retrieval agent performs a similarity search of the one or more vector stores to retrieve a set of passages from the plurality of reference documents based on similarity of encodings of the inference input and encodings of passages in the plurality of reference documents. The database system generates a linguistic prompt for an LLM having a context including the inferences and passages and applies the LLM to the linguistic prompt to generate a natural language explanation of the one or more inferences.

The invention discloses a data storage method for an artificial intelligence learning mode, and relates to the technical field of computer data storage, and the method comprises the steps: 1, merging a multi-source perception stream into blocks in real time at the edge through monotone serial number writing, so as to provide a replayable time sequence; 2, asynchronous erasure coding is executed on the blocks, Merkel roots are calculated and written into a local cache, and dual guarantee of loss tolerance and integrity is achieved; 3, pushing slices and roots to object storage in sequence according to a network, and calling a time travel interface to solidify an incremental snapshot; 4, the cloud end monitors a snapshot hash event, a serial number chain is written through differential scanning, a gap is reconstructed through slices, an index is refreshed, and continuous consistency is kept; 5, the training process generates a Merkel proof online verification sample, and damaged data are immediately interpolated and repaired and an audit chain is recorded; and step 6, after training is finished, generating a leatherwise list and a frozen root, asynchronously cleaning redundant slices, updating a version table, and finally forming single-fingerprint traceable cost archiving.

A heterogeneous storage-oriented hot and cold data management method and system, wherein hot and cold data are managed by means of the following steps: at table granularity, storing data from each table into an independent data bucket in the form of object storage, each data bucket comprising a plurality of partitions, the data being randomly distributed in each partition, and the consistency of replicas of the partitioned data being ensured by means of a distributed consensus algorithm; collecting statistics about the access frequencies of the partitions in one data bucket; when the access frequency of one partition reaches a set threshold, adding a new replica to a target storage medium by means of a membership change in the distributed consensus algorithm; and when data synchronization between the new replica and an old replica is completed, detecting the old replica. In this way, the migration of the partitioned data on different storage media is realized, and data cooling and heating are implemented. The method and system can maximize the use efficiency of high-performance storage media, ensure the access efficiency of users, and can reduce the costs of data migration and maintenance.

The invention provides an abnormity early warning security system and method based on cross-modal semantic retrieval, and relates to the technical field of public security, and the system comprises a video collection and frame sampling module which is used for extracting key frame images from a real-time monitoring video stream according to a preset frame interval and storing the key frame images to an object storage system; the target detection module adopts a YOLO model to execute instance-level target detection on the extraction frame, and generates a structured detection result containing a target position, a category and confidence; the semantic vector storage and structured warehousing module packages the standardized metadata of the image frame and the semantic vector generated by the CLIP into structured data, and stores the structured data into a high-performance vector database; and the semantic retrieval and reverse query module receives a natural language query instruction to generate a semantic vector, and realizes cross-modal retrieval of historical monitoring images through vector similarity matching. According to the invention, a set of exception security system with real-time perception capability, semantic understanding capability and cross-modal retrieval capability is constructed.

The invention provides a component configuration method which can be applied to the technical field of cloud computing and the technical field of big data. The method comprises the steps of obtaining configuration information used for configuring a target interface component from a server side based on an interface request; packaging the interface request into an asynchronous data acquisition object, and storing the asynchronous data acquisition object as a global shared instance; registering a callback processing function based on the global shared instance for the target interface component; and responding to the configuration information returned by the server side and triggering the callback processing function, and executing component rendering on the target interface component based on the configuration information.

Embodiments are directed to a file system that include object stores. An object store for write requests may be provided. Write ahead log (WAL) entries that include data blocks may be generated. A WAL object may be generated based on the WAL entries and stored in the object store. An in-memory overlay may be updated to associate the data blocks with the WAL object. A checkpoint operation may be executed to: generate an index object that includes index entries that associate other data blocks with data objects stored in the object store; update the index object to include index entries that associate the data blocks with the WAL object; store the updated index object in the object store; update the in-memory overlay to remove the association of the data blocks and the WAL object and update the in memory WAL to remove records of successfully checkpointed WAL objects.

The invention relates to the technical field of research and development management, can be applied to business scenes of financial science and technology, medical health and the like, and discloses an anomaly analysis method, device, equipment and medium based on a variation test. Generating a test case through the semantic perception model; selecting a variation strategy by using a state-driven model according to the service state, and generating a variation test case based on the access control strategy; executing the variation test case in a distributed environment, and collecting system resource index data and request execution monitoring data; and inputting the collected data into the abnormal prediction model, and outputting an abnormal behavior prediction result. Through combination of semantic perception generation, state-driven selection and anomaly prediction, coverage of a test case on protocol dependence and a service state is improved, and the accuracy of anomaly detection of the object storage system is remarkably improved.

The invention relates to the technical field of data processing, and discloses a data storage optimization method and system for a solid state disk. The method comprises the steps of performing pattern recognition on small object metadata through multi-dimensional access feature extraction, performing five-level popularity level division based on a flash memory perception popularity evaluation algorithm, performing multi-level storage region division on an SSD, performing intelligent resource allocation according to popularity levels and storage regions, and realizing dynamic adjustment processing through adaptive load monitoring. The technical problem of how to carry out intelligent hot-cold separation and dynamic hierarchical storage on the small object metadata to reduce the SSD storage cost in a massive small object storage environment generated by a cloud native application is solved.

The invention provides a life cycle deletion optimization method for a distributed object storage system, which comprises the following steps of: S1, detecting the business volume of a cluster, starting a pre-scanning operation when the business volume is lower than a preset threshold value, and identifying an object to be deleted according with a life cycle rule; s2, respectively writing the to-be-deleted object information of different dates into corresponding omaps, and storing omap data in a disk according to a preset packaging frequency; s3, when the life cycle expires, directly reading the omap of the corresponding date, skipping a real-time scanning step, and executing batch deletion operation; and S4, recording statistical information such as the deletion quantity scanned every day on the xattr of the barrel, so that management software performs data trend analysis and operation and maintenance decision. According to the embodiment of the invention, the life cycle deletion efficiency can be improved, the metadata operation pressure is reduced, and the cluster storage space utilization rate is optimized.

The invention relates to the technical field of big data storage, and discloses a multi-level data storage and mixed query method and system, a storage medium and equipment, and the method comprises the steps: collecting original buried point data, carrying out real-time stream processing, storing the processed real-time data in a hot storage layer, carrying out timing offline batch processing on historical buried point data, and storing the processed real-time data in a hot storage layer; converting into an optimized storage format and storing in an object storage system; an extended metadata management system is constructed, and structure information, partition information and access frequency information of data in the object storage system are automatically extracted and managed; executing query, and reducing the data scanning amount through a query optimization strategy; automatically migrating data among different storage layers through a dynamic cold and hot data layering mechanism; a unified query interface is provided, a query request is distributed to a corresponding real-time processing system or an offline query system based on an intelligent routing strategy, and a query result is cached and returned, so that high-performance and low-cost storage and second-level query of data are realized.

The invention discloses a log processing method and device, electronic equipment and a computer program product. Relates to the field of financial science and technology, and comprises the following steps: receiving a log query request, and extracting log generation time and a log tag of a target log file from the log query request, the log tag being used for indicating a file type of the target log file; generating a query condition according to the log generation time and the log tag, and obtaining a storage path of the target log file from a query index associated with the object storage system according to the query condition to obtain a target storage path; and extracting a target log file from the object storage system based on the target storage path, and sending the target log file to the client. Through the method and the device, the technical problem that the query efficiency is low when the target log file is acquired from the computing cluster in related technologies is solved.

The invention discloses a distributed file management system and method, and belongs to the technical field of computer software, the system adopts a tree hierarchical structure to organize files, and the system comprises an application layer used for receiving file operation requests from a plurality of independent clients, and each request carries a unique application identifier; the file system layer comprises a configuration module, an operation module, a metadata management module, a day module, a timed task module and a gateway module, and the request is routed to a corresponding logic storage partition in the data layer according to an application identifier; the data layer comprises a database cluster and an object storage cluster, and performs persistent storage and management of logic isolation on metadata and file data according to application identifiers; and the infrastructure layer provides an extensible server and network resources. Based on a distributed architecture, load balancing and efficient access are realized through data storage partitioning, a metadata management and index technology and a client load balancing technology, and the expandability, fault tolerance and performance of the system are improved.

The invention relates to the technical field of data processing, in particular to an enterprise data asset management method, system and device based on cloud data and a storage medium, an activity sensing engine is deployed to collect indexes such as data access frequency, business key weight and cross-department calling popularity, and the indexes are processed into multi-dimensional activity factors through a rule engine; and dividing hot, warm and cold grades after calculating scores. Hot data is controlled by a trusted domain and stored in a high-IOPS cloud disk, warm data generates role self-adaptive desensitization view retention standard cloud disk, and cold data starts full mask and manual approval transfer object storage. And if the cold data is overdue, starting compliance review and physically erasing the cold data. According to the method, dynamic collaboration of data value evaluation, security policy and storage scheduling is realized, traditional static management contradictions are solved, full-life-cycle intelligent management of data is achieved, the value utilization rate is improved, the resource cost is optimized, and compliance risks are reduced.

The invention provides a server-free document processing and real-time reporting method and system, relates to the technical field of data processing, and starts from responding to the operation that a user uploads a source document to an object storage server, and routing an event to a pre-subscribed document processing server-free function. And after the function is triggered, obtaining a source document, performing analysis and data extraction on the source document to generate structured data, and storing the structured data in a database. The event bus then routes the event to a pre-subscribed report-generating server-less function. And automatically generating a notice containing an access link and pushing the notice to a specified user or terminal. According to the method, automation, real-time performance and flexibility of the whole process from document uploading to report generation are achieved, and through tight combination of event driving and a server-free architecture, the processing efficiency, the resource utilization rate and the system response speed are improved.

The described technology pertains to a distributed computing environment, specifically implementing a streaming protocol on a scalable object storage service. The technical problem addressed is the high computing resource demand of distributed streaming platforms. The solution involves separating content from location metadata, allowing centralized, scalable storage of metadata. Brokers store content in object storage services based on specific metrics, such as fast read / write times or greater storage capacity. A background platform remaps metadata sequences based on logical timestamps, optimizing data retrieval. This system enhances the efficiency of computing resources by reducing the load on brokers and improving data access speeds. A use can be in cloud computing systems for efficient content streaming and storage management.

In some implementations, a storage node receives data processing instructions from a compute node of a network architecture. The storage node detects a Directed Acyclic Graph (DAG) within the data processing instructions. The DAG includes a plurality of data processing operations arranged within the DAG according to an execution. The data processing operations further include user-defined function (UDF) information associated with a UDF. The storage node may retrieve a first data object from an object store of the distributed storage node based on object keys information within the plurality of data processing operations. The storage node may execute the plurality of data processing operations using the retrieved first data object to generate a DAG execution result, the executing of the plurality of data processing operations based on the execution order. The storage node may communicate the DAG execution result to the compute node in response to the data processing instructions.

Managing access to stored objects includes receiving, by a storage OS in a computing system, an access key identifier (ID) and access key associated with a user of an object storage service (OSS); determining, by a directory service, whether the user is a member of a group authorized to access a storage object in at least one of a network accessible storage (NAS) volume and an OSS bucket, based at least in part on the access key ID; in response to the user being a group member, attempting to authenticate the user by the storage OS with the directory service based at least on the access key ID and the access key; and in response to the user being authenticated by the directory service, allowing, by the storage OS, access by the user to the storage object stored in at least one of the NAS volume and the OSS bucket.

A unified security agent manager plugin within a virtual machine compute instance manages at least one agent installed within the compute instance of a cloud environment. The plugin periodically receives agent inventory information, where the agent inventory information identifies (i) a plurality of platform types of the agent, and (ii) for each platform type, one or more deployable versions of the agent. The plugin selects a platform type from the plurality of platform types. The plugin compares the one or more deployable versions corresponding to the selected platform type with a version of the agent currently installed in the compute instance. If the version currently installed in the compute instance is older than the one or more deployable versions, the plugin fetches an agent object corresponding to a deployable version from an object storage repository, and updates the version currently installed in the compute instance to the fetched deployable version.

The invention relates to the field of data processing, in particular to a scientific and technological achievement data batch arrangement method based on OCR and semantic processing. Comprising the following steps: scanning paper scientific and technological achievements to obtain an archive image; preprocessing and segmenting the archive image, and converting the archive image into a digital document; extracting corresponding metadata from the digital document; storing the digital document and the corresponding metadata into a database as a paper scientific and technological achievement in a structured data form; generating a knowledge graph according to the digital document and the corresponding metadata, and storing the knowledge graph in a graph database in a knowledge graph form; storing the original scanning copy of the paper scientific and technological achievement into the storage object; correlating the contents of the same paper scientific and technological achievements in the database, the graph database and the object storage. Through the multi-mode OCR recognition, dynamic semantic annotation, intelligent data analysis and knowledge fusion technology, full-process automation of scientific and technological achievement data from collection, processing to storage is achieved.

The invention discloses a virtual machine data backup method and device of a cloud computing management platform, a program product, equipment and a medium, and relates to the technical field of computers. The method comprises the following steps: scanning file system metadata of a disk mirror image file of a virtual machine snapshot, screening out a current change file from the disk mirror image file, and determining a current change data block needing to be backed up in an incremental backup mode by comparing a target data block of the current change file with a corresponding data block of reference backup data; the reference backup data is a basic backup data set for incremental comparison; performing hierarchical compression processing on the current changed data block; and constructing a target storage directory in the object storage service according to the virtual machine identifier and the backup task identifier associated with the cloud computing management platform, and backing up the compressed data to the target storage directory. According to the invention, the backup data size and backup time are optimized, and the limitation of low backup function efficiency of the cloud computing management platform is effectively improved.

The invention relates to a distributed high-availability code warehouse management method and system and a medium, and relates to the technical field of version control and code management.The method comprises the steps that a pre-configured external load balancer, a plurality of GitLab Web nodes, an internal load balancer, a PostgreSQL database cluster, a Redis cluster, a Gitaly cluster and object storage or file storage are obtained; carrying out load balancing on the plurality of GitLab Web nodes through the external load balancer; performing load balancing on the internal connection of the GitLab application program through the internal load balancer; the data of the GitLab Web and the data of the GitLab Pracect are stored through the PostgreSQL database cluster, and the data of the GitLab Web and the data of the GitLab Pracect are And providing access to a Git warehouse by using the Gitaly cluster. According to the invention, high availability, high performance and expandability of code warehouse management are realized through a distributed architecture and redundancy configuration.

The invention discloses a multi-source document management method and device based on knowledge construction and fusion storage, and relates to the technical field of document management, and the method comprises the steps: receiving a multi-source document to an object storage system, and recognizing the document type; performing document analysis and structure extraction according to the document type; extracting pictures in the image-text mixed content, uploading the pictures to an object storage system, generating a mapping dictionary, and inserting picture marks in a text part of the mapping dictionary; carrying out structured processing on contents of the table key value pairs to extract summaries and abstracts; performing standardization processing and semantic slicing on the processed image-text mixed content, table key value pair content and / or plain text content to generate knowledge fragments; constructing a knowledge extracting questions from knowledge fragments; the knowledge fragments are stored in a first index, and the questions and IDs of the corresponding knowledge fragments are stored in a second index; generating vectors for each knowledge fragment and question, embedding and writing the vectors into a vector database; and performing document management based on the knowledge base. The document management efficiency is improved.

The invention provides a multi-type storage cluster dynamic nano-management method based on a cloud platform, and belongs to the technical field of cloud platform storage resource management, a storage pool is used as a minimum management unit, a configuration center provides a unified nano-management entrance and is associated with a platform initialization process, and initialization of standardized parameters such as a storage cluster IP, an account and the like is automatically completed; after management, block storage, file storage or object storage types and exclusive parameters thereof are configured for the storage pool according to business requirements, and available storage pools are screened according to binding relations when storage services are created, so that resource mismatching is avoided. Bidirectional synchronization is carried out between the operation and maintenance resource management module and the storage module through a full life cycle by using an equipment SN number, so that state consistency and traceability are realized; expansion of newly added storage pools, unbinding of the storage pools and resource recovery are supported, and the resource utilization rate and the operation and maintenance efficiency are improved.

The invention relates to the technical field of aviation field multi-source heterogeneous data retrieval, and discloses an aviation field multi-source heterogeneous data deep retrieval engine based on a large model, which comprises a front end and API gateway, a LangGraph agent, a retrieval layer, a data access and treatment layer, an object storage module and a monitoring and operation layer, according to the aviation field multi-source heterogeneous data deep retrieval engine based on the large model, unified retrieval can be conducted on multi-source heterogeneous data such as patents, standards and literatures, a user does not need to conduct respective retrieval in different data sources, the retrieval efficiency is greatly improved, and through unified data access and cooperative work of a governance layer and a retrieval layer, the retrieval efficiency is greatly improved. The barriers among different data sources are broken through, and keywords and evaluation strategies conforming to industry characteristics are generated in combination with the thesaurus, the technical system and the knowledge graph in the aviation field. During retrieval sorting, factors such as industry priorities are considered, so that the retrieval result better meets the actual requirements of the aviation field, and the retrieval accuracy is improved.

The invention provides a file data migration method and device, equipment, a storage medium and a program. The method comprises the following steps: receiving object storage access information, wherein the object storage access information is used for requesting migration of target file data; according to the object storage access information, constructing mapping information between a file local directory of the file storage system and an object storage bucket of an object storage cluster; according to the file path under the local directory of the file, generating a target object identifier in the object storage cluster, the target object identifier being generated by converting a hierarchical structure of the file path; and migrating the target file data from the file storage system to an object storage cluster based on the mapping information and the target object identifier. According to the method, the file storage flexibility can be improved.

Techniques for implementing a scalable key value storage in a distributed storage separate the storage of the collection of data objects from the storage of the index corresponding to the collection. According to an implementation, a database service may receive a request to create a collection of data objects in an object storage. A schema may be specified for the collection. The database service may generate a unique identifier (ID) corresponding to the collection and create the collection in the object storage. The objects in the collection and the associated schema may be further written to an object table and a schema table, respectively. The database service may further validate the schema and extract one or more indexable fields from the schema. The database service may send a request to a database search service to create an index for the collection in an index storage.

The invention provides a distributed data storage method, device and equipment and a readable storage medium, and the method comprises the steps: responding to an event created by a storage target, obtaining metadata of the storage target, and submitting the metadata to a key-value pair database in a key-value pair manner; a distributed file system is called to split the specific file into distributed storage objects, and the distributed storage objects obtained through splitting are sent to object storage equipment of a distributed storage cluster; and in response to an event of storage target change, retrieving the corresponding key value pair and modifying the metadata recorded by the corresponding key value pair according to the change condition. Through the technical scheme of the specification, the metadata is stored by the key-value pair granularity, the problem of directory-level read-write amplification is relieved, single file metadata update only needs to modify the corresponding key-value pair, overall directory rewriting is avoided, and IO consumption and network delay are reduced in combination with a write-in merging mechanism of an embedded database and the transparent object splitting capability of a distributed file system.

The invention discloses a composite file management system and method. According to the system, a three-in-one collaborative architecture is formed by integrating samba service, storage service and SaaS service. Wherein the samba service adopts a five-layer modular design, the network communication module supports an SMB / CIFS protocol to perform data interaction with a multi-platform terminal, the request processing module realizes high-concurrency access control through an asynchronous queue mechanism, and the service logic module is internally provided with file version management, authority verification and log auditing functions. The file storage calling module is in butt joint with a heterogeneous storage system in an adapter mode and supports mixed deployment of a distributed file system and object storage, and the SaaS system calling module is deeply integrated with third-party application service through RESTful API to achieve cross-system file circulation and metadata synchronization. According to the method, transparent scheduling of local storage and cloud storage is achieved through a unified file operation interface, and an enterprise-level file management solution capable of being elastically expanded is constructed.

The invention discloses a computer vision processing system based on machine learning, and the system comprises a dynamic topology management module which constructs and maintains a dynamic knowledge graph of a camera network in real time through a graph neural network technology, and carries out the topology self-adaption; the intelligent search module is used for dynamically selecting an optimal monitoring node for query based on a reinforcement learning technology; the multi-modal detection center is used for integrating multi-modal data, carrying out flame / smoke detection by combining an OfficientDet-D7 model with an optical flow method, integrating living body detection in a face recognition model trained in an MS1M-ArcFace data set based on an ArcFace framework to resist attacks, and carrying out multi-target tracking and cross-camera re-recognition by utilizing a DeepSORT algorithm; the distributed early warning system is used for triggering graded early warning through a dynamic threshold engine and carrying out context-aware false alarm suppression and key alarm priority response; the data persistence layer is used for storing structured data by adopting MongoDB fragmentation clusters and storing original video streams by adopting MinIO object storage; and the edge calculation node is used for locally executing lightweight calculation.