541 results about "Data warehouse" patented technology

The invention relates to an enterprise-level simulation knowledge graph construction method based on multi-modal data integration, and belongs to the technical field of knowledge graphs. The method comprises the following steps: integrating structured data, semi-structured data and unstructured data through a multi-modal data warehouse; performing knowledge extraction on the semi-structured data and the non-structured data to obtain entities and relationships, and performing knowledge fusion; storing the fused entities and relationships by using a graph database, and constructing a simulation knowledge graph; a vector database is embedded in combination with a simulation knowledge graph, semantic extension search is realized through multi-modal joint search, similar cases are searched through a simulation result graph, and a simulation scheme comparison matrix is automatically generated. The multi-modal data is effectively integrated, the comprehensiveness and accuracy of knowledge graph construction are improved, more powerful, efficient and intelligent support is provided for simulation analysis of enterprises, and the enterprises can be assisted in rapidly making scientific decisions in complex and changeable business scenes.

Systems and methods for translating natural language to SQL on a custom enterprise data warehouse powered by Generative AI. With an embodiment of the present invention, a natural language question may be converted to a meaningful and accurate database query, e.g., SQL query, relevant to tables existing in an enterprise data warehouse. An embodiment of the present invention is directed to a comprehensive approach of transforming a natural language query to a focused SQL query using domain specific data models across firmwide metadata systems and data systems. In response to a user query, an embodiment of the present invention performs metadata analysis, targeted data retrieval and then SQL generation. An embodiment of the present invention may apply data warehousing standards and guidelines followed in the enterprise and provide a plug-and-play type architecture and solution that is scalable to large warehousing and other systems.

The invention discloses an AI data warehouse full-link blood relationship tracking method and device, and relates to the technical field of data processing. The method comprises the following steps: obtaining structured log data with business terms and entity tags; establishing a knowledge graph according to the structured log data and the business document; generating blood relationship metadata according to the knowledge graph and the data job execution log; constructing a blood relationship cognition map according to the blood relationship metadata, the data operation log and the business document; generating a governance strategy vector according to the blood relationship cognitive map, the system real-time index and the service SLO, and sending the governance strategy vector to an execution engine, so that the execution engine carries out processing according to the governance strategy vector; and obtaining a governance strategy execution result, user feedback information of the governance strategy execution result and a system log generated in a governance process. According to the method, complex semantics and causal relationships behind data operation can be captured.

Reducing amounts of data ingested into a data warehouse, including: determining a duration associated with a subject described by event data provided to a data ingestion pipeline; determining, based on the duration, an interval for providing next event data; and providing, based on the interval, the next event data.

The invention relates to the technical field of data optimization, and discloses a financial data warehouse intelligent SQL (Structured Query Language) optimization method based on a large model, which comprises the following steps: S1, SQL statement analysis and preprocessing: carrying out word segmentation, grammatical analysis and logic structure analysis on an SQL statement submitted by a user, and identifying keywords, table structures and query conditions in query; s2, large model semantic understanding and optimization suggestion: performing semantic understanding and logical reasoning on the analyzed SQL statements by using a large model, and automatically generating optimization suggestions, the optimization suggestions including index recommendation, query rewriting and execution plan optimization; and S3, generating and selecting a dynamic execution plan. According to the method, the advantages of a large model in the aspects of natural language processing and intelligent decision making are fully utilized and combined with the specific requirements of the financial data warehouse, and the execution efficiency of SQL query and the overall performance of the system are remarkably improved; the query response time of the financial data warehouse can be effectively shortened, and the resource utilization rate is improved.

The invention discloses a manufacturing enterprise data center system and method based on a large model, and the system comprises a data collection layer which is used for constructing a multi-source heterogeneous data collection system; the data storage layer is used for constructing a data warehouse system comprising multiple layers and realizing data lake storage; the data processing layer and the service layer are used for deploying a model and integrating an anomaly detection module, a multi-modal fusion engine and a prediction model; the service layer is used for an interface group, integrating a Drive / ClickHouse query engine and a Redis cache, and supporting multi-condition combined query and millisecond response; deploying a visual service module and an RBAC / ABAC authority control system to realize data grading sharing; and the application layer is used for developing a production optimization application, a supply chain management application and a CRM application. Through a uniform data acquisition interface and an efficient data acquisition tool, mass data inside and outside an enterprise can be integrated, data islands are broken, centralized management and sharing of the data are realized, and availability and value of the data are improved.

The invention provides a domain data management method and system based on an artificial intelligence platform, and the method comprises the steps: obtaining financial transaction data through a data warehouse, carrying out the preprocessing, and extracting transaction metadata and semantic association thereof through a natural language technology; and classifying the metadata by adopting a hierarchical clustering algorithm, and constructing a transaction data consanguinity map. And extracting time sequence and spatial features from the atlas, calculating a node interaction index by using a pre-configured data interaction prediction model, and marking a high-heat node. And extracting high-dimensional transaction association features based on the graph neural network model, and identifying abnormal transaction data. And marking an abnormal node in the blood relationship map, and tracking and positioning an origin node of abnormal data. According to the method, multiple advanced technologies are comprehensively utilized, and rapid and accurate risk identification can be realized in the face of massive financial transaction data.

The invention relates to the technical field of lithium battery safety monitoring, and discloses a lithium battery pack thermal runaway early warning system based on multi-mode sensing. The system comprises multi-source sensing data acquisition, thermal field feature tensor construction, thermal field reconstruction and thermal coupling association network generation. The multi-source sensing data acquisition module acquires multi-dimensional heterogeneous data from a temperature sensor, a voltage and current monitoring unit, a gas component detector and an acoustic emission sensor, and generates a standardized data bin through timestamp alignment and missing value compensation; the thermal field feature tensor construction module extracts features such as temperature gradient and electrochemical response from the data bin in a multi-scale manner, and constructs a tensor in combination with time continuity; the thermal field reconstruction module generates association diagrams according to the feature space-time distribution and fuses the association diagrams into a lithium battery pack three-dimensional thermal field reconstruction map; and the thermal coupling association network generation module extracts a feature vector cluster, calculates an entropy weight value, and generates a network according to a high-entropy node space adjacency relationship. According to the system, multi-dimensional data fusion is realized, and the thermal runaway evolution law can be comprehensively described.

The invention discloses a data warehouse intelligent construction method and device based on a large model. The invention provides a data warehouse intelligent construction method and device based on a large model, and belongs to the technical field of big data processing. The method comprises the following steps: a user registers to-be-collected business system data source information through a data source registration entry, and submits the information as a data processing task; receiving a data processing task through a task executor, calling a large model to carry out feature analysis on a data table, and identifying a table type, a service attribute and an acquisition parameter; executing data acquisition according to an analysis result, including identifying a temporary table, determining an acquisition period, selecting an increment field and the like; performing intelligent exploration on the collected data, and identifying field features and data quality; performing data conversion based on the probing result, wherein the data conversion comprises automatically binding a business rule, associating a data standard and generating a conversion rule; and finally, executing data fusion, intelligently matching the target table and the field mapping relation through a large model, generating a fusion rule, and executing fusion.

The invention discloses an equipment data acquisition and query optimization method based on a large model enabling ERP-MES system, and relates to the technical field of integrated enterprise resource planning and monitoring management systems, and the method comprises the following steps: carrying out the dynamic adaptive data acquisition and preprocessing of equipment: employing an edge node to deploy a lightweight noise detection model, identifying an electromagnetic interference mode in sensor data collected by the equipment in real time, and dynamically compensating and correcting by combining environment temperature and humidity data; according to the method, an edge node lightweight noise detection model is combined with environmental data compensation and correction, and a BERT large model is utilized to analyze a protocol document design middleware, so that accurate acquisition and efficient processing functions of dynamic self-adaptive data of equipment are realized; preprocessed data are mapped and fused through a multi-modal large model to construct a unified data warehouse, and a mapping strategy is optimized through reinforcement learning, so that the equipment data quality is improved, and the data fusion and utilization efficiency is enhanced while heterogeneous data fusion is achieved.

The invention provides a report generation and optimization method and system based on an offline data warehouse. The system comprises a data extraction layer, a data loading layer, a data cleaning layer, a data integration layer, a model data layer, a report mart layer, a report generation layer and a report optimization layer. The method comprises the following steps: S1, extracting data; s2, data cleaning; s3, integrating the data; s4, ensuring data quality; s5, storing model data; s6, a report market is constructed; and step S7, report generation. A layered architecture is adopted, independent operation of each module of the system is ensured, and expansion and maintenance are facilitated. Data extraction, cleaning and integration processes are simplified by introducing an automatic tool, and redundant calculation is reduced by optimizing SQL and solidifying logic. Bottom logic is optimized for large-scale data volume and complex query, and report generation performance is improved. And monitoring and management functions of the whole life cycle of the data flow are provided, and data quality and report accuracy are ensured.

The invention relates to a big data-based intelligent electricity sale quantity prediction method, which comprises the following steps of S1, collecting multi-dimensional data related to electricity sale quantity, and establishing a unified data warehouse S2, constructing multi-dimensional features; s3, using a feature importance algorithm and a mutual information algorithm to quantify the relation between the features and the target variables, and rejecting redundant features; s4, constructing an intelligent electricity sale quantity prediction model based on the LSTM-TFT hybrid model; and S5, processing the real-time streaming data by using Spark Streaming, generating online features, deploying a trained model LSTM-TFT hybrid model, and obtaining a prediction result through TF-Serving. According to the method, the time sequence and the external features are dynamically modeled, so that accurate prediction of the electricity sale quantity is realized.

The invention discloses an urban traffic intelligent management method based on multi-source data, and relates to the technical field of traffic data management, and the method comprises the steps: collecting multi-source traffic data, firstly carrying out the preprocessing, then carrying out the matching and integration, and building a unified traffic data warehouse; carrying out batch processing and stream processing on warehouse data by applying a distributed computing framework and a stream processing technology; on the basis of multi-source data, various traffic models are constructed and optimized, and a visual tool is developed to visually display an analysis result; by combining real-time traffic data, a model prediction result and historical data, traffic signals are dynamically adjusted, route planning and navigation services are provided for users, and routes and scheduling of public traffic are optimized; a traffic monitoring center is constructed, and real-time monitoring and timely early warning are carried out based on the traffic signal dynamic adjustment result and the real-time traffic data; establishing an emergency plan; and establishing a data sharing platform, and sharing the related traffic data. The system is used for monitoring, analyzing, managing and applying the traffic data.

The invention discloses a computer resource allocation management system and method based on data analysis, and relates to the technical field of computer resource management.The method comprises the steps that data are collected in real time, and historical data are collected; constructing a spatio-temporal data warehouse based on the real-time data and the historical data; extracting static features and dynamic features of the task, and combining the static features and the dynamic features to generate a high-dimensional task feature vector; constructing a resource semantic knowledge graph, and defining a computing resource entity, a relationship and a constraint rule; combining the spatio-temporal data warehouse and the resource semantic knowledge graph to construct a demand prediction model, and outputting probability distribution prediction of future resource demands; and in combination with real-time data, calculating resource use efficiency and a load balancing index in real time, and dynamically adjusting a resource allocation decision. The demand prediction model is constructed based on the spatio-temporal data warehouse and the resource semantic knowledge graph, resource allocation is carried out actively, resource shortage or idle is avoided, and the resource utilization rate is improved.

The invention discloses a cable fault accurate positioning system based on an intelligent sensor system, and relates to the technical field of power fault positioning, and the cable fault accurate positioning system comprises a cloud monitoring center which is in communication connection with the following modules: a signal acquisition and processing module which is used for deploying the intelligent sensor system along a cable, operation environment data and traveling wave signals in the cable operation process are obtained, and the signals are preprocessed and stored in a cloud data warehouse. According to the invention, the front band-pass filter and the self-adaptive trap filter are combined to filter out high-frequency harmonic waves and specific-frequency interference signals, and the differential sampling technology is adopted to counteract common-mode interference, so that the signal-to-noise ratio of the signals is remarkably improved, a system can accurately capture and process weak traveling wave signals caused by a high-resistance grounding fault in a complex environment, and the fault detection accuracy is improved. Therefore, the fault positioning precision is greatly improved, the positioning error caused by signal interference is reduced, and the determination of the fault position is more reliable and accurate.

Embodiments of the present disclosure provide a data processing method, apparatus based on data Lake and data Warehouse integration, and an electronic device. The method includes: pre-storing at least two protocol files; receiving a first metadata acquisition request; determining a target protocol file for processing the first metadata acquisition request from the at least two protocol files according to an engine type of the data processing engine which sends the first metadata acquisition request; and parsing, based on the target protocol file, the first metadata acquisition request and acquiring metadata corresponding to the first metadata acquisition request from the metadata storage space. Therefore, for at least two sets of external protocols, one set of metadata is stored, accordingly, the data storage capacity can be reduced; with the setting of one set of metadata, the metadata accessing rights management can be unified, and the security of data can be improved.

The invention relates to a query method based on query auxiliary information. The method comprises the following steps: acquiring a natural query statement of a user and user information; obtaining query auxiliary information according to the user information; determining candidate query fields associated with the natural query statement by matching query fields in a preset constellation data model; the natural query statement, the candidate query fields, the query auxiliary information and a preset first task processing text are input into a first text processing model, the query intention of the natural query statement is understood through the first text processing model in combination with the query auxiliary information, and the candidate query fields conforming to the query intention are determined as target query fields. And further, determining at least one target data table only comprising the target query field from the constellation data model, and inputting the natural query statement and the target data table into a query instruction generation model to generate a query instruction. And finally, based on the query instruction, querying from the data warehouse to obtain a query result of the natural query statement.

The invention relates to the technical field of big data processing, in particular to an auditing big data information collection method, which comprises the following steps of S1, identifying and classifying various data sources; s2, making a corresponding acquisition strategy according to the features of the various data sources identified in the S1; s3, acquiring data from each data source in real time according to the acquisition strategy formulated in S2; s4, storing the data acquired in the S3 into a data warehouse according to a preset data structure; s5, performing quality evaluation on the data stored in the data warehouse; s6, information in the data warehouse is periodically updated through a data synchronization mechanism, efficient collection, standardized processing and intelligent quality control of multi-source data are achieved through a collection strategy and an innovative data quality evaluation method, and the automation degree of data processing and the reliability of auditing analysis are remarkably improved.

The invention relates to the technical field of medical health big data analysis, and discloses a disease trend analysis and early warning system and method based on big data, and the method comprises the steps: 1, obtaining multi-source medical data through a hospital information platform, carrying out the preprocessing of the data, building a correlation mapping relation, and forming a standardized data warehouse; 2, constructing a spatio-temporal data cube, and forming a three-dimensional relation graph of time, space and diseases; 3, establishing a disease analysis model according to the multiple influence factors, and performing dynamic analysis on the disease trend and the high-incidence diseases based on the disease analysis model; and 4, according to an analysis result, identifying a disease critical area, generating area early warning information and feeding back the area early warning information to a corresponding terminal. According to the method, the disease monitoring accuracy and real-time performance are effectively improved, the early warning capability is improved, resource configuration is fully integrated, and the disease response efficiency is improved.

The invention discloses a water conservancy data analysis method and device based on a self-attention mechanism and a medium, and relates to the technical field of water conservancy. The method comprises the following steps: constructing a water conservancy time sequence data warehouse, and establishing a water conservancy source data access mechanism to access to-be-processed water conservancy time sequence data; performing standardized preprocessing on the to-be-processed water conservancy time sequence data to obtain standard time sequence data; inputting the standard time sequence data into a pre-trained water conservancy data analysis model, and calculating attention weight distribution between different time steps and different variables through a self-attention mechanism unit in the water conservancy data analysis model; according to the attention weight distribution, performing weighted summation on the standard time sequence data, generating a spatial-temporal feature representation vector, inputting the spatial-temporal feature representation vector into a gating circulation unit in the water conservancy data analysis model, and executing spatial-temporal information fusion by updating a gate and resetting a gate; and performing water conservancy data trend prediction based on the fused features, and outputting flood prevention scheduling decision support data.

The invention relates to a travel destination recommendation method and system based on big data and computer equipment, and the method comprises the steps: collecting the original data of travel history, browsing history and social media activities of a user from multiple channels, and constructing a user behavior data warehouse after preprocessing; key features reflecting user travel modes and preferences are extracted from a user behavior data warehouse, user feature vectors are constructed according to the key features, and a candidate destination recommendation list is generated through a mixed recommendation model. And constructing a feature vector according to geographic positions, climate conditions and resource endowment data of the candidate destinations, obtaining an optimal access time period through an ecological influence evaluation model, and matching with a recommendation list to screen out a proper destination. And finally, in combination with multiple preferences of the user, a complete travel plan is generated by using a reinforcement learning algorithm, and the purposes of accurate recommendation, ecological protection and real-time dynamic adjustment are achieved.

According to the WeChat operation automation comprehensive service platform provided by the invention, by integrating the firewall architecture module, the unified management module and the user information integration module, safe and efficient management of enterprise WeChat data and seamless integration with enterprise CRM are realized. The firewall architecture module adopts a proxy server technology and intelligent traffic analysis, monitors and filters potential malicious traffic in real time, and ensures data security; the unified management module stores and manages application attributes, main data and public data of enterprise WeChat in a centralized manner by using a structured data warehouse technology, and optimizes a data management process; the user information integration module realizes seamless integration of enterprise WeChat user information and CRM user information through an SSO technology and a data mapping engine; and the operation efficiency and the data management capability of an enterprise on the WeChat platform are greatly improved.

The invention discloses a multi-dimensional operation data analysis and method based on a BI system, and relates to the technical field of multi-dimensional operation data analysis, and the system comprises a data integration module which is in butt joint with real-time data of a business system through a BI platform; the data logic processing module adopts a streaming computing framework to compute and update data in real time, and the timed task layer configures periodic batch tasks and executes historical data aggregation, trend analysis and cross-module association computation; the data warehouse module is used for storing the processed data and carrying out classified storage according to business dimensions to form a unified business data warehouse; the distributed computing engine module is used for scheduling and distributing tasks; and the visual report module is used for generating a multi-dimensional operation analysis report. Through real-time data integration, streaming computation and timed task configuration, efficient data processing and personalized requirements are supported, large-scale parallel computation and load balancing are supported, and disaster recovery capacity is enhanced.

The invention discloses an ancient building three-dimensional reconstruction method based on multi-modal data fusion, and relates to the technical field of data processing, and the method comprises the steps: obtaining multi-modal data of an ancient building and historical culture, and constructing a data warehouse for storing the obtained multi-modal data of the ancient building and historical culture; performing semantic alignment processing on the multi-modal data of the historic building and the historical culture to obtain multi-modal fusion feature information; inputting the multi-modal fusion feature information into a completely trained artificial intelligence neural network model for depth calculation and estimation judgment, and generating depth map data; and obtaining point cloud data based on the depth map data, and performing three-dimensional model reconstruction according to the obtained point cloud data and a three-dimensional reconstruction technology to complete virtual reality implementation of the ancient building. Based on an artificial intelligence multi-modal data fusion technology, the invention creates a multi-modal data fusion three-dimensional reconstruction virtual reality method for an ancient building and historical culture thereof based on artificial intelligence.

The invention relates to the technical field of data analysis, and discloses a business analysis and decision support method and system, and the method comprises the steps: collecting purchase-related business data from a plurality of data sources, and carrying out the preprocessing; storing the preprocessed data in a centralized data warehouse, and performing management based on a preset data governance strategy; performing multi-dimensional analysis on the structured data based on a statistical analysis model and a machine learning algorithm, and generating a key performance indicator KP I and a trend prediction result; providing a customized visual display interface according to a user role, and presenting the KP I and a trend prediction result through an interactive instrument panel; and generating a purchase optimization suggestion based on the analysis result, evaluating the influence of different purchase strategies through analogue simulation, and outputting corresponding decision support information. The system effectively solves the problems of weak analysis capability, insufficient decision support and the like in an existing purchase decision system, and is suitable for multiple application scenes such as enterprise purchase management and the like.

The invention relates to the technical field of big data, and provides a data verification method and device in big data cluster migration, equipment and a medium, and the method comprises the steps: obtaining a source data cluster and a target data cluster; dividing the source data cluster and the target data cluster into a plurality of data layers according to a standard data warehouse layering strategy, and binding different data comparison strategies for each data layer; establishing a fingerprint baseline table and constructing a sampling model feature library; performing data fragmentation processing on each data layer, and controlling each data layer to execute a data comparison task in parallel; and if any data layer of the target data cluster and any data layer of the source data cluster are detected to have difference, recording difference data and performing data restoration. According to the method, the source data cluster and the target data cluster are subjected to hierarchical parallel fragmentation comparison, a plurality of fragments of each layer are subjected to data comparison tasks in parallel at the same time, different comparison strategies are bound to the data layers, the comparison process can be achieved, and the overall data processing efficiency is improved.

The invention discloses a fire-fighting spatial-temporal trajectory data processing and off-line data warehouse construction method based on a stream batch fusion architecture, and relates to the field of digital fire fighting, and the core steps of the method are as follows: after multi-source data are collected by an edge gateway, high throughput distribution and storage are realized by Kafka; real-time optimization such as track cross repair and offset correction is completed through Flink streaming calculation; performing offline batch processing on the historical data by the Spark to generate an analysis result; generating an optimal rescue path through a distributed Dijkstra algorithm in combination with road topology, real-time traffic and fire fighting truck parameters; the method comprises the following steps: realizing mixed storage by adopting ElasticSearch secondary index and HBase total storage; based on hot and cold data hierarchical management, intelligent applications such as fire-fighting commanding and dispatching, battle comment and review and the like are supported. Practice verifies that the method can shorten the fire control time of the main urban area by 60%, significantly improves the fire emergency response capability, and is suitable for large-scale fire track data scenes.

The invention relates to the technical field of nuclear power data asset automatic management, in particular to a nuclear power digital twinning data warehouse structure system, which comprises a data processing layer, and is characterized in that the data processing layer reads original data of a nuclear power industrial internet platform, performs data standardization processing on the original data to obtain standard data, and sends the standard data to the nuclear power industrial internet platform; constructing a nuclear power industry theme library according to the standard data; the data integration layer performs data coding mapping, data classification and data fusion on the data of the nuclear power industry theme library to form a thematic library; the semantic enhancement layer marks the data of the nuclear power industry theme library and the data in the thematic library formed by the data integration layer, and constructs a structured feature tag library for the data associated with feature tags; the analysis application layer reads data of the feature tag library, and constructs a digital twinborn algorithm library and a digital twinborn sample library through algorithm analysis and sample training. The system supports equipment operation and maintenance from regular maintenance to predictive maintenance based on bottom support and four-layer progressive architecture of a nuclear power industrial internet platform.

The invention belongs to the technical field of intelligent collaborative decision-making, and particularly relates to a multi-dimensional evaluation and integrated optimization method based on intelligent collaborative decision-making, which comprises the following specific steps: S1, data processing: constructing a unified data acquisition platform, accessing multi-source heterogeneous data including structured data, semi-structured data and unstructured data, data cleaning, conversion and standardization technologies are adopted, the data quality problem is eliminated, data islands are broken, and enterprise-level or industry-level data warehouses are established; and S2, analysis processing: performing mining analysis on the multi-dimensional data by using an artificial intelligence algorithm, and constructing a multi-dimensional evaluation index system. According to the system and the method, a unified data acquisition platform is constructed, multi-source heterogeneous data is accessed, data cleaning, conversion and standardization technologies are adopted, an enterprise-level or industry-level data warehouse is established, data barriers among departments can be broken, and efficient integration and sharing of data are realized.

The invention provides a correlation analysis method for technological parameters in a wafer processing process and related equipment. The correlation analysis method comprises the following steps: determining target parameters needing correlation analysis in the wafer processing process; according to the target parameter, acquiring all other process parameters belonging to the same wafer batch as the target parameter and corresponding parameter values in the data warehouse, and packaging into a data set; inputting the data set into a machine learning algorithm model, and calculating the correlation between the target parameter and other process parameters through a machine learning algorithm; and determining key process parameters influencing the target parameters according to a correlation calculation result. The correlation between the target parameter and other process parameters is automatically calculated and identified through a machine learning algorithm. In this way, the key process parameters influencing the target parameters can be effectively screened out, objective data support and decision basis are provided for the process defect problem caused by the common influence of multiple factors, and the difficulty of fault analysis and problem positioning is reduced.