2093 results about "Big data processing" patented technology

The present invention relates to the technical field of big data processing, and discloses a multi-source heterogeneous data fusion and processing method based on big data, including the following steps: data preprocessing, cleaning, complementing and feature extraction of multi-source heterogeneous data, providing standardized and high-quality input for subsequent processing; dynamic knowledge graph modeling, constructing a dynamic knowledge graph based on the semantic relationship of multi-source data, and generating a unified semantic representation; cross-modal alignment and deep fusion, semantically aligning multi-modal data, and generating a unified comprehensive representation through deep fusion of features; real-time stream processing, dynamically updating and processing real-time updated stream data to achieve real-time fusion of data; adaptive optimization, dynamically adjusting fusion strategy parameters, and improving the adaptability and robustness of the fusion strategy. The present invention realizes efficient fusion, real-time processing and semantic consistency expression of multi-source heterogeneous data through dynamic knowledge graph modeling, cross-modal alignment, real-time stream processing and adaptive optimization.

The invention provides a reimbursement data management system and method based on AI artificial intelligence, and the system comprises a multi-modal data collection and fusion module which is used for providing an access reimbursement application entrance; the auditing and compliance judgment module is used for invoice authenticity identification, reason rationality identification, reimbursement item classification and automatic auditing; the approval process optimization module is used for learning and optimizing a reimbursement approval process and automatically simplifying or increasing approval levels according to reimbursement application features; the analysis and prediction module is used for performing multi-dimensional analysis on the reimbursement data based on a big data processing framework and a data mining algorithm and generating a visual report and result; and the data storage and block chain module is used for constructing a distributed and non-tampering data storage account book by adopting a block chain technology, and executing a reimbursement process automation rule by utilizing a block chain intelligent contract. The efficiency, quality and safety of reimbursement data management are comprehensively improved, and enterprise financial management is promoted to develop towards the intelligent and refined direction.

The invention provides an AI-based post competency and recruitment post matching method, and relates to the technical field of big data processing, and the method comprises the steps: obtaining multi-source heterogeneous data, and extracting dominant skill entities and implicit ability entities of candidates; constructing a candidate ability evolution graph; analyzing a post dynamic demand by combining a time sequence prediction model, and forming a dynamic post portrait including a current demand, a hidden demand and a future evolution demand; calculating the structural similarity between a candidate map and a post portrait through a map neural network matching algorithm, and generating and sorting comprehensive matching scores in combination with adaptive dimensions such as core capability difference, future integrating degree and team integration potential; team cooperative effect simulation is introduced, the influence of candidate addition on a team capability structure and a cooperative network is predicted, and the model is optimized through a closed-loop feedback mechanism; the talent recognition accuracy and coverage range are remarkably improved, prospective strategic talent matching is achieved, and the matching efficiency is improved.

The invention discloses a document information input method and system based on big data processing, and belongs to the technical field of document information processing. According to the method, deep risk feature screening is performed on document data by combining big data mining and a knowledge graph, potential sensitive elements can be accurately identified, risk categories and influence ranges are analyzed based on association rules, automatic early warning and real-time risk notification are realized, a dynamic rule knowledge base is adopted, multi-dimensional compliance verification is supported, and the risk assessment efficiency is improved. Service constraint conditions can be updated in real time, the flexibility and adaptability of rules are ensured, the accuracy and timeliness of compliance check are improved, pattern matching and multi-level check are performed on compliance data objects through an intelligent check process and a recheck rule set trained by big data, the preciseness and reliability of data check are ensured, and the data check efficiency is improved. Through comprehensive quantitative calculation of the business complexity, the risk coefficient and the timeliness priority, the task scheduling weight parameter is generated, the receipt auditing path is optimized, and the auditing efficiency is improved.

The present invention proposes a vector database retrieval method and system, which relates to the technical field of big data processing and intelligent information retrieval. The method includes preprocessing high-dimensional multimodal data to obtain a dimension reduction feature vector; constructing a cluster center based on the dimension reduction feature vector; establishing an inverted file and an intra-cluster plane index based on the cluster center, compressing the inverted file and the intra-cluster plane index to obtain a compressed inverted file and an intra-cluster plane index; obtaining a user's query vector, using the inverted file to screen candidate clusters for the query vector, introducing a hierarchical navigable small-world graph in the intra-cluster plane index, performing an intra-cluster search on the cluster cluster, and obtaining a vector database retrieval result. The present invention effectively takes into account the efficiency and accuracy of high-dimensional data retrieval, and improves the retrieval efficiency and accuracy.

The invention discloses a multi-modal data fusion method and system based on a large-model intelligent agent, belongs to the technical field of artificial intelligence, big data processing and intelligent agents, and aims to solve the technical problem of how to effectively integrate data of different modalities such as texts, images and voices by using the large-model intelligent agent. According to the technical scheme, the method comprises the steps that data collection and preprocessing are conducted, specifically, various modal data of texts, images and voice are collected through a web crawler, an API interface, a camera and microphone equipment, the collected data are preprocessed, the preprocessed multi-modal data are obtained, and the data quality is ensured; feature extraction and mapping: extracting corresponding modal features from the preprocessed multi-modal data through CNN and Transform models, mapping the different modal features to the same space, and combining the aligned features to form comprehensive feature representation; carrying out multi-modal fusion processing; and performing intelligent decision and feedback.

The invention relates to the technical field of big data processing, and provides a mass data storage optimization method which comprises the following steps: analyzing target data according to access frequency and / or access time of the data, and dividing the target data into hot data, warm data and cold data; aiming at the hot data, the temperature data and the cold data, performing compression processing by adopting corresponding compression algorithms; before the target data is stored, performing duplicate removal on the target data according to the fingerprint of the target data; monitoring the use condition of the memory, the access frequency of the target data and the performance of the compression algorithm in real time, and deploying system resources according to a monitoring result; and setting a multi-level cache, and caching different types of target data to the multi-level cache according to the divided types of the target data. According to the technical scheme, mass data can be efficiently stored and rapidly accessed, and the storage space utilization rate and the overall system performance are remarkably improved.

The invention discloses a self-adaptive intelligent symbiotic data ecosystem and a cooperative implementation method thereof. The system comprises an intelligent agent ecosystem module, a self-adaptive domain knowledge graph module, a self-adaptive workflow engine module, a CLID configuration system module, a man-machine cooperation interface module, a data processing pipeline module and an innovation trigger module. Through cooperation of diversified intelligent agents, the dynamic knowledge graph, the self-adaptive workflow and CLID multi-dimensional configuration, high-flexibility and self-adaptive data processing and analysis capacity is achieved. The system can automatically construct a complex workflow, perform multi-modal data fusion, and integrate human judgment and AI capabilities at key points, thereby providing an innovative solution. The method is suitable for complex and dynamic big data processing and analysis scenes, and the data processing efficiency and innovation ability are remarkably improved.

The invention discloses an enterprise scientific and technological achievement adaptation method based on big data accurate retrieval and query, and relates to the technical field of big data processing and knowledge maps, and the method comprises the steps: constructing an enterprise technical field knowledge map based on a standardized multi-modal feature matrix, receiving incremental technical data in real time by adopting an Apache Flink streaming processing framework, and carrying out the real-time retrieval and query of the enterprise technical field knowledge map. Dynamically updating a graph node relationship, removing expired nodes through a pruning algorithm, and outputting a dynamic knowledge graph with a timestamp; and receiving a query request of a user, extracting a query vector by utilizing the cross-modal embedding model, retrieving a Top-N candidate node list through a dynamic knowledge graph, calculating a comprehensive similarity score in combination with a graph relation weight, and outputting a sorted technical achievement list. Multi-source heterogeneous data is mapped to a unified semantic space through a cross-modal embedding model, and the intelligent level and the actual application effect of scientific and technological achievement adaptation are comprehensively improved in combination with construction and updating of a dynamic knowledge graph.

The invention relates to the technical field of intelligent recognition, in particular to an intelligent tumor recognition system and method based on big data. The system comprises a tumor big data processing module, a tumor identification model construction module, a tumor multi-source feature fusion module and a tumor classification identification mapping module, and can acquire corresponding tumor multi-modal medical image data and patient pathological clinical data through butt joint and perform image registration and filtering cleaning processing. Tumor multi-mode standard image data and patient clinical standard data are obtained; constructing a corresponding tumor recognition model; performing feature analysis and big data fusion splicing on the tumor multi-modal standard image data and the clinical standard data of the patient to obtain a multi-source fusion feature vector of the tumor of the patient; and inputting the multi-source fusion feature vector of the tumor of the patient into a constructed tumor identification model to carry out tumor classification and identification, thereby outputting a medical image tumor identification result. According to the invention, accurate recognition of tumors can be realized.

The invention provides a deep reinforcement learning optimization algorithm based on an adaptive encryption framework, and the algorithm comprises ten steps: data input preprocessing, data sensitivity evaluation, system resource state acquisition, state vector construction, strategy decision, encryption execution, calculation reward, model updating, result output, and loop optimization. The method aims at solving the problems that encrypted data processing efficiency is low and safety is difficult to consider. According to the method, intelligent optimization of the encryption strategy is realized through the data sensitivity evaluation module, the adaptive encryption strategy adjustment module, the deep reinforcement learning optimization module, the parallel processing module, the resource scheduling module and the like. On the premise that the data security is ensured, the processing efficiency of the system is remarkably improved, and the method is suitable for scenes such as cloud computing, big data processing and the Internet of Things which need to process a large amount of sensitive data.

The invention relates to the technical field of agricultural big data processing, and discloses a winter wheat yield prediction method based on XLSTM and assimilation crop growth model, comprising the following steps: data acquisition and data preprocessing; extracting a winter wheat planting area; performing leaf area index inversion; correlation analysis and feature extraction; establishing a crop growth model based on a WOFOST model; performing data assimilation based on ensemble Kalman filtering; establishing a time sequence data set; establishing a yield prediction model based on the extended long-short-term memory neural network; and model prediction: inputting the current remote sensing image and the meteorological data of the future time period into a yield prediction model, and outputting a yield prediction value of the winter wheat in the target area. According to the method, by integrating and fusing multi-source data and combining complex models and algorithms, the problem that fusion of the multi-source data and the complex relation between the multi-source data and the yield are not fully considered due to the fact that only a single feature is concerned or the models are not fully considered is solved.

An intelligent design method for a support scheme of a rock burst section in a TBM tunnel, relating to the field of tunnel engineering. The intelligent design method for the support scheme of the rock burst section in the TBM tunnel comprises: determining a surrounding rock class on the basis of TBM mainframe operation data, surrounding rock feedback evaluation, and a tunnel engineering geological survey report, and using a rock burst intensity assessment method to predict a rock burst grade; and on the basis of the surrounding rock class and the rock burst grade, using a computer deep learning technology to determine an initial surrounding rock support method. The method effectively avoids the impact of subjective factors, actively utilizes TBM mainframe operation data and surrounding rock feedback evaluation, utilizes big data processing and an artificial intelligence technology to propose an intelligently designed support scheme for a rock burst section in a TBM tunnel, and can provide an efficient support scheme for field construction workers for a rock burst section in a tunnel.

The invention discloses a private domain data-based user portrait generation method and a private domain data-based content generation and touch method, and relates to the technical field of big data processing and pushing. Analyzing user behavior characteristics and interest preferences based on the collected data; marking static tags for the corresponding users based on the basic attributes; according to the behavior characteristics and interest preferences, constructing a primary dynamic tag; through user behavior data collected in real time, a time decay weighting algorithm is adopted to improve recent behavior weight and filter abnormal click data, a time sequence cross attention network is adopted to model a behavior feature sequence, interest preference and probability are output, and a dynamic threshold value is set to trigger dynamic label updating; and if the key behavior data of the user is collected, recalculating the dynamic tag based on the key behavior data and performing full-amount refreshing. According to the invention, a data-content-touch real-time linkage closed-loop system is constructed.

The invention relates to the technical field of computers, in particular to a task scheduling optimization analysis method and system based on big data. The method comprises the following steps: acquiring task related data, task system resource use data and task execution state data in real time in a corresponding distributed big data processing scene among a server, a storage device and a client; performing big data feature mining analysis and task scheduling clustering analysis by using a Hadoop distributed framework to generate task scheduling cluster groups corresponding to similar resource requirements; performing residual execution duration calculation and task emergency accounting on the task calculation nodes corresponding to the distributed big data processing scene to obtain task node scheduling emergency degrees; and performing scheduling priority determination and task scheduling optimization analysis on the corresponding task calculation nodes to generate a task dynamic optimization scheduling scheme. According to the method, the rationality of distributed task resource scheduling can be improved.

The invention discloses a data warehouse intelligent construction method and device based on a large model. The invention provides a data warehouse intelligent construction method and device based on a large model, and belongs to the technical field of big data processing. The method comprises the following steps: a user registers to-be-collected business system data source information through a data source registration entry, and submits the information as a data processing task; receiving a data processing task through a task executor, calling a large model to carry out feature analysis on a data table, and identifying a table type, a service attribute and an acquisition parameter; executing data acquisition according to an analysis result, including identifying a temporary table, determining an acquisition period, selecting an increment field and the like; performing intelligent exploration on the collected data, and identifying field features and data quality; performing data conversion based on the probing result, wherein the data conversion comprises automatically binding a business rule, associating a data standard and generating a conversion rule; and finally, executing data fusion, intelligently matching the target table and the field mapping relation through a large model, generating a fusion rule, and executing fusion.

The invention discloses an off-site derivative cross-border supervision intelligent adaptation system based on heterogeneous data fusion, and relates to the technical field of big data processing. A data acquisition module is set to obtain off-site derivative transaction data, and an original data set is generated; setting a data structured processing module to perform semantic analysis on the original data set by adopting a financial text-oriented text model to generate a transaction information table, setting a transaction type classification module to judge whether cross-border transactions are involved or not according to transaction types in the transaction information table, and if the cross-border transactions are involved, setting a cross-border compliance evaluation module to perform cross-border compliance evaluation on the cross-border transactions. Performing dynamic rule matching on the transaction information table to generate a compliance report; otherwise, a risk prediction module is set, risk prediction is conducted on the transaction information table through a pre-trained transaction graph neural network model, a risk early warning signal is generated, and corresponding early warning measures are executed. The compliance cost is reduced, and the risk management and control capability is improved.

The invention relates to the technical field of photovoltaic power generation, in particular to a flexible photovoltaic intelligent monitoring and management system and method based on the Internet of Things, multi-source heterogeneous data are comprehensively collected through deployed multiple types of Internet of Things sensor nodes, the data are uploaded to a cloud platform after being cleaned and standardized through edge nodes, a big data processing architecture integrated with flow and batch is adopted, and the intelligent monitoring and management system and method based on the Internet of Things are established. The method comprises the following steps: performing real-time analysis and state judgment on a real-time data stream, performing deep batch processing and feature mining on historical data, extracting high-order features such as a performance attenuation trend and an abnormal mode, fusing real-time and historical features, and realizing comprehensive scoring of a health state of a component and accurate prediction of residual life by utilizing a machine learning model. And based on an evaluation result and a preset knowledge base, automatically generating a differentiated precise operation and maintenance instruction, and issuing and executing the differentiated precise operation and maintenance instruction to form closed-loop management. According to the invention, the monitoring depth and breadth of the flexible photovoltaic system are effectively improved, the conversion from passive alarm to active predictive maintenance is realized, and the operation reliability of the system is significantly enhanced.

The invention relates to the technical field of big data processing and analysis, in particular to a cross-border payment path optimization system and method based on multi-dimensional data analysis. Payment transaction data and exchange rate data are collected, a multi-dimensional payment path evaluation model is constructed from the angles of payment cost, payment timeliness, payment risk, payment availability and the like, and each payment path is scored in real time; constructing a payment path optimization model, dynamically selecting an optimal path and performing intelligent scheduling; a machine learning algorithm is introduced, an abnormal transaction mode is automatically identified from mass payment data, and a real-time anomaly detection and risk prevention and control strategy is established; according to the method provided by the invention, the success rate of cross-border payment can be improved, the cost is reduced, the payment time is shortened, and the payment risk is prevented, so that the cross-border payment service quality and the user experience are comprehensively improved.

The invention discloses an operation site safety management and control method and system based on a multi-modal large model, and the method comprises the steps: collecting site violation data, and carrying out the preprocessing, and obtaining multi-modal data; constructing a multi-modal large model; carrying out training and optimization on the multi-modal large model by utilizing the preprocessed multi-modal data; and using the optimized multi-modal large model to realize operation site safety management and control. According to the invention, a visible light image, a safety violation knowledge base and other modal data are fused, and a comprehensive safety monitoring and management platform is constructed by using a large model and a big data processing technology, so that real-time monitoring, early warning and intelligent response of potential safety hazards of a complex operation site are realized, and the safety management level of the operation site is improved.

The invention provides an intelligent building construction management system and method based on BIM scene construction, and relates to the technical field of building construction management. In the construction process, the management system collects mass data in real time by means of sensing layers spread all over key nodes of a construction site; the method comprises the following steps of: transmitting operation parameters of construction equipment, operation dynamics of constructors, consumption rate of materials and change of a field environment to a data processing center, accurately positioning a low-efficiency waiting problem existing in a material transfer link through deep analysis of a big data processing engine and an artificial intelligence algorithm library, recommending an optimization scheme, and improving the efficiency of material transfer. According to the method, the hoisting sequence is adjusted, unnecessary working hour loss is reduced, by means of a BIM model platform, a construction plan is presented in a visual mode, and the management system can rapidly generate and recommend and flexibly deploy construction team personnel, optimize the construction process sequence, ensure construction coherence, reduce construction delay and shorten the construction period.

The invention discloses a multi-source heterogeneous data fusion communication system and method, relates to the technical field of big data processing, and effectively solves the limitation of a traditional single data source monitoring mode by integrating various heterogeneous data sources and collecting and preprocessing data in real time when the system runs. Through the data fusion and feature extraction module, the system can comprehensively capture the multi-dimensional state features of the equipment and obtain a feature data set Fset (t), thereby providing accurate data support for subsequent anomaly detection and prediction. Under the combined action of the anomaly detection model and the prediction model, the system can discover potential faults in time and give out early warning in advance, and meanwhile, a self-adaptive regulation and control strategy is generated to prevent further deterioration of the anomaly. In addition, through a feedback and self-adaptive optimization module, the accuracy of fault prediction and the effectiveness of a regulation and control strategy are improved, the maintenance cost is reduced, and the efficiency and the safety are remarkably improved.

The invention discloses a fall risk real-time assessment system and method based on big data, and relates to the field of risk assessment, and the system comprises a data collection and preprocessing module, a big data processing and integration module, a feature extraction and selection module, a fall risk assessment module and a real-time monitoring and early warning module which are sequentially connected and cooperatively work. Wherein the data acquisition and preprocessing module is responsible for acquiring data from a plurality of sources and converting the data into digital signals; the big data processing and integration module is responsible for secure transmission, convergence and preprocessing of data; the feature extraction and selection module is responsible for extracting features related to the falling risk from the preprocessed data; the tumble risk assessment module is responsible for real-time assessment of tumble risks based on the extracted features; and the real-time monitoring and early warning module is responsible for triggering an early warning signal according to the falling risk assessment result. According to the invention, accurate and real-time evaluation of the falling risk, accurate and timely output of the evaluation result and rapid response of the early warning signal are realized.

The invention relates to the technical field of transformer monitoring, and discloses a comprehensive online transformer monitoring system which comprises a sensor layer, an edge computing layer, a cloud platform layer and a communication module. The cloud platform layer comprises a data warehouse, a big data processing engine, a comprehensive diagnosis engine, a model training engine and a system management module, the edge calculation layer is in communication connection with the cloud platform layer through a communication module, the comprehensive diagnosis engine integrates a deep neural network and a knowledge graph inference engine, and cross validation of data driving and knowledge guiding is achieved; a model training engine utilizes historical data and online incremental data to continuously optimize a model, and a dynamic knowledge graph updates a fault rule through a real-time diagnosis result and expert feedback, so that the system can adapt to novel faults and complex working conditions, and the diagnosis accuracy and adaptability are greatly improved.

The invention discloses an information management method based on a big data processing technology, and the method comprises the steps: S1, data collection and pre-processing optimization: S1.1, multi-source data cleaning and pre-processing: automatically detecting and correcting errors, missing and abnormal values in data through an integrated intelligent data cleaning technology, such as a machine learning algorithm (such as abnormal detection and missing value filling); the data quality is ensured, data standardization and format unification tools are adopted, heterogeneous data from different sources are converted into a unified format, and follow-up processing is facilitated. The problems of data privacy, quality, real-time performance, complexity and compliance in a traditional big data method are solved by enhancing data quality management, improving data security, optimizing storage and processing architecture and introducing modes such as real-time processing and intelligent analysis. Meanwhile, the expandability and the fault-tolerant capability of the system ensure the high efficiency and the stability of the system in a large-scale data environment, and a more intelligent, safer and more efficient information management solution is provided for enterprises.

The invention relates to the technical field of big data processing, in particular to an express logistics path planning system based on topology analysis. The invention discloses an express logistics path planning system based on topology analysis. The system comprises a node load feature extraction module, a space-time cross correlation screening module, a road contour dynamic correction module, a node optimization sorting module and a path continuous connection reconstruction module. According to the method, the node load change trend is extracted, the node state evolution is dynamically mastered, the load change response precision is improved, the accuracy of correlation identification between the nodes is enhanced based on space-time joint screening, the road boundary is dynamically corrected, the geometric precision of the topological structure is improved, and the node sorting is optimized in combination with the passing time and the communication continuity; connection errors are inhibited through continuous offset detection, the path communication precision is improved, comprehensive control over node dynamics, space-time correlation, road structures and path precision is achieved on the whole, and the real-time performance, flexibility and accuracy of logistics path planning are enhanced.

The present application relates to the technical field of big data processing, and discloses a risk behavior identification method and apparatus, a storage medium, and a computer device. The method comprises: acquiring access traffic data of sensitive data, and on the basis of service categories to which access users corresponding to the access traffic data belong, classifying the access traffic data to obtain sensitive data access data of a plurality of service categories; on the basis of the sensitive data access data of each service category, updating model parameters of a risk assessment model of the service category; by means of the risk assessment model of the service category and on the basis of the sensitive data access data of the service category, calculating a risk behavior assessment value of each access user under the service category; and on the basis of the risk behavior assessment value of each access user under the service category, obtaining a risk behavior identification result of each access user. The method can improve the adaptability, the practicability and the assessment accuracy of risk behavior assessment in service scenarios, and provides visual assessment results.

The invention relates to a big data processing technology, in particular to an intelligent data labeling method and system. The intelligent data labeling method comprises the following steps: preprocessing public opinion data; semantic deduplication of the historical sample library is carried out, and the sample library is updated; an intelligent auxiliary labeling method based on a historical work order library and a large model; model updating and sample increment updating; compared with the prior art, the method has the advantages that a multi-dimensional data cleaning method is adopted, the quality of the to-be-labeled data is improved, and interference of repeated semantic information is reduced through semantic duplicate removal; potential values of similar work orders in historical work orders are mined, a large model cue word project used for auxiliary labeling is constructed, an available auxiliary labeling cue word template is provided, and the cue word construction and auxiliary labeling process is opened; a PDCA thought is adopted, and a data-annotation-model-work order closed-loop management and PDCA mechanism spiral data quality method is constructed.

The invention discloses an electronic product personalized marketing method based on a big data accurate portrait, and relates to the technical field of big data processing, and the method comprises the following steps: multi-source data collection and cross-domain fusion processing: obtaining a multi-source data set related to an electronic product from each platform of the Internet by using a web crawler and a streaming processing engine, the multi-source data set is subjected to cleaning classification and alignment fusion processing to generate multi-modal data, and then the multi-modal data is updated in real time to dynamically reflect data changes, so that a foundation is laid for subsequent data analysis; according to the method, multi-source data is collected by using a web crawler and a streaming processing engine, quantitative analysis is performed on the data by using a Hofscode culture dimension model and a Maslow demand level theory, a user portrait is constructed by using a distributed data flow engine Apache Flink to dynamically update a label, and a GAN (GAN Adversarial Generative Network) anomaly detection and evaluation analysis method is applied, so that the user portrait is obtained. And comprehensive and accurate data acquisition and processing are realized.

The invention relates to the technical field of data analysis and calculation, in particular to a distributed scheduling method based on tasks, nodes and link resources, which is applied to a solution for task scheduling and management in a large-scale distributed system. The method is particularly suitable for cloud computing, big data processing and distributed storage systems which need to process mass data and high-concurrency requests. And dynamically adjusting resource allocation by using real-time monitoring data and a prediction algorithm in combination with task priorities and resource requirements. Compared with traditional static resource allocation, a dynamic resource allocation mechanism is adopted, resource allocation can be intelligently adjusted in real time according to task requirements and node resource conditions, and therefore the resource utilization rate and the task execution efficiency are improved.