742 results about "Data warehouse" patented technology

The systems and methods described herein provide intelligent people analytics from generative artificial intelligence. In one embodiment, the system: receives a prompt related to people analytics from a client device associated with a user; generates an embedding representation of the received prompt using a generative AI system including one or more generative AI models; performs a similarity search using the generated embedding representation to identify similar prompts that have been submitted before; obtains an executable expression for responding to the received prompt; executes the executable expression using a data warehouse comprising one or more data sources to obtain a response to the received prompt; determines a type of response based on the nature of the received prompt; generates a response output based on the determined type and the response to the received prompt; and provides the response output to the client device associated with the user.

The invention relates to an enterprise-level simulation knowledge graph construction method based on multi-modal data integration, and belongs to the technical field of knowledge graphs. The method comprises the following steps: integrating structured data, semi-structured data and unstructured data through a multi-modal data warehouse; performing knowledge extraction on the semi-structured data and the non-structured data to obtain entities and relationships, and performing knowledge fusion; storing the fused entities and relationships by using a graph database, and constructing a simulation knowledge graph; a vector database is embedded in combination with a simulation knowledge graph, semantic extension search is realized through multi-modal joint search, similar cases are searched through a simulation result graph, and a simulation scheme comparison matrix is automatically generated. The multi-modal data is effectively integrated, the comprehensiveness and accuracy of knowledge graph construction are improved, more powerful, efficient and intelligent support is provided for simulation analysis of enterprises, and the enterprises can be assisted in rapidly making scientific decisions in complex and changeable business scenes.

Systems and methods for translating natural language to SQL on a custom enterprise data warehouse powered by Generative AI. With an embodiment of the present invention, a natural language question may be converted to a meaningful and accurate database query, e.g., SQL query, relevant to tables existing in an enterprise data warehouse. An embodiment of the present invention is directed to a comprehensive approach of transforming a natural language query to a focused SQL query using domain specific data models across firmwide metadata systems and data systems. In response to a user query, an embodiment of the present invention performs metadata analysis, targeted data retrieval and then SQL generation. An embodiment of the present invention may apply data warehousing standards and guidelines followed in the enterprise and provide a plug-and-play type architecture and solution that is scalable to large warehousing and other systems.

The invention discloses an AI data warehouse full-link blood relationship tracking method and device, and relates to the technical field of data processing. The method comprises the following steps: obtaining structured log data with business terms and entity tags; establishing a knowledge graph according to the structured log data and the business document; generating blood relationship metadata according to the knowledge graph and the data job execution log; constructing a blood relationship cognition map according to the blood relationship metadata, the data operation log and the business document; generating a governance strategy vector according to the blood relationship cognitive map, the system real-time index and the service SLO, and sending the governance strategy vector to an execution engine, so that the execution engine carries out processing according to the governance strategy vector; and obtaining a governance strategy execution result, user feedback information of the governance strategy execution result and a system log generated in a governance process. According to the method, complex semantics and causal relationships behind data operation can be captured.

Reducing amounts of data ingested into a data warehouse, including: determining a duration associated with a subject described by event data provided to a data ingestion pipeline; determining, based on the duration, an interval for providing next event data; and providing, based on the interval, the next event data.

The invention relates to the technical field of data optimization, and discloses a financial data warehouse intelligent SQL (Structured Query Language) optimization method based on a large model, which comprises the following steps: S1, SQL statement analysis and preprocessing: carrying out word segmentation, grammatical analysis and logic structure analysis on an SQL statement submitted by a user, and identifying keywords, table structures and query conditions in query; s2, large model semantic understanding and optimization suggestion: performing semantic understanding and logical reasoning on the analyzed SQL statements by using a large model, and automatically generating optimization suggestions, the optimization suggestions including index recommendation, query rewriting and execution plan optimization; and S3, generating and selecting a dynamic execution plan. According to the method, the advantages of a large model in the aspects of natural language processing and intelligent decision making are fully utilized and combined with the specific requirements of the financial data warehouse, and the execution efficiency of SQL query and the overall performance of the system are remarkably improved; the query response time of the financial data warehouse can be effectively shortened, and the resource utilization rate is improved.

The invention discloses a manufacturing enterprise data center system and method based on a large model, and the system comprises a data collection layer which is used for constructing a multi-source heterogeneous data collection system; the data storage layer is used for constructing a data warehouse system comprising multiple layers and realizing data lake storage; the data processing layer and the service layer are used for deploying a model and integrating an anomaly detection module, a multi-modal fusion engine and a prediction model; the service layer is used for an interface group, integrating a Drive / ClickHouse query engine and a Redis cache, and supporting multi-condition combined query and millisecond response; deploying a visual service module and an RBAC / ABAC authority control system to realize data grading sharing; and the application layer is used for developing a production optimization application, a supply chain management application and a CRM application. Through a uniform data acquisition interface and an efficient data acquisition tool, mass data inside and outside an enterprise can be integrated, data islands are broken, centralized management and sharing of the data are realized, and availability and value of the data are improved.

The invention discloses a safe and rapid storage method and system for image data of an imaging department, and relates to the technical field of medical image data processing. The method comprises the following steps: 1, designing a distributed storage system, and selecting and configuring storage nodes; 2, selecting and updating an encryption algorithm, and implementing file-level and block-level encryption; 3, realizing accurate data deduplication by utilizing a machine learning technology, and selecting a proper compression algorithm; 4, constructing an index system, and establishing a data warehouse and a data lake by using big data analysis and machine learning technologies; 5, establishing an image data backup and recovery mechanism; and 6, constructing a data integrity checking and monitoring system. According to the invention, by adopting the technologies of distributed storage, encryption, data deduplication, compression, big data analysis, machine learning, backup and recovery, data integrity verification, monitoring and early warning and the like, efficient storage and safety protection of image data are realized.

The invention provides a domain data management method and system based on an artificial intelligence platform, and the method comprises the steps: obtaining financial transaction data through a data warehouse, carrying out the preprocessing, and extracting transaction metadata and semantic association thereof through a natural language technology; and classifying the metadata by adopting a hierarchical clustering algorithm, and constructing a transaction data consanguinity map. And extracting time sequence and spatial features from the atlas, calculating a node interaction index by using a pre-configured data interaction prediction model, and marking a high-heat node. And extracting high-dimensional transaction association features based on the graph neural network model, and identifying abnormal transaction data. And marking an abnormal node in the blood relationship map, and tracking and positioning an origin node of abnormal data. According to the method, multiple advanced technologies are comprehensively utilized, and rapid and accurate risk identification can be realized in the face of massive financial transaction data.

The invention relates to the technical field of lithium battery safety monitoring, and discloses a lithium battery pack thermal runaway early warning system based on multi-mode sensing. The system comprises multi-source sensing data acquisition, thermal field feature tensor construction, thermal field reconstruction and thermal coupling association network generation. The multi-source sensing data acquisition module acquires multi-dimensional heterogeneous data from a temperature sensor, a voltage and current monitoring unit, a gas component detector and an acoustic emission sensor, and generates a standardized data bin through timestamp alignment and missing value compensation; the thermal field feature tensor construction module extracts features such as temperature gradient and electrochemical response from the data bin in a multi-scale manner, and constructs a tensor in combination with time continuity; the thermal field reconstruction module generates association diagrams according to the feature space-time distribution and fuses the association diagrams into a lithium battery pack three-dimensional thermal field reconstruction map; and the thermal coupling association network generation module extracts a feature vector cluster, calculates an entropy weight value, and generates a network according to a high-entropy node space adjacency relationship. According to the system, multi-dimensional data fusion is realized, and the thermal runaway evolution law can be comprehensively described.

The invention discloses a data warehouse intelligent construction method and device based on a large model. The invention provides a data warehouse intelligent construction method and device based on a large model, and belongs to the technical field of big data processing. The method comprises the following steps: a user registers to-be-collected business system data source information through a data source registration entry, and submits the information as a data processing task; receiving a data processing task through a task executor, calling a large model to carry out feature analysis on a data table, and identifying a table type, a service attribute and an acquisition parameter; executing data acquisition according to an analysis result, including identifying a temporary table, determining an acquisition period, selecting an increment field and the like; performing intelligent exploration on the collected data, and identifying field features and data quality; performing data conversion based on the probing result, wherein the data conversion comprises automatically binding a business rule, associating a data standard and generating a conversion rule; and finally, executing data fusion, intelligently matching the target table and the field mapping relation through a large model, generating a fusion rule, and executing fusion.

The invention discloses an equipment data acquisition and query optimization method based on a large model enabling ERP-MES system, and relates to the technical field of integrated enterprise resource planning and monitoring management systems, and the method comprises the following steps: carrying out the dynamic adaptive data acquisition and preprocessing of equipment: employing an edge node to deploy a lightweight noise detection model, identifying an electromagnetic interference mode in sensor data collected by the equipment in real time, and dynamically compensating and correcting by combining environment temperature and humidity data; according to the method, an edge node lightweight noise detection model is combined with environmental data compensation and correction, and a BERT large model is utilized to analyze a protocol document design middleware, so that accurate acquisition and efficient processing functions of dynamic self-adaptive data of equipment are realized; preprocessed data are mapped and fused through a multi-modal large model to construct a unified data warehouse, and a mapping strategy is optimized through reinforcement learning, so that the equipment data quality is improved, and the data fusion and utilization efficiency is enhanced while heterogeneous data fusion is achieved.

The invention provides a report generation and optimization method and system based on an offline data warehouse. The system comprises a data extraction layer, a data loading layer, a data cleaning layer, a data integration layer, a model data layer, a report mart layer, a report generation layer and a report optimization layer. The method comprises the following steps: S1, extracting data; s2, data cleaning; s3, integrating the data; s4, ensuring data quality; s5, storing model data; s6, a report market is constructed; and step S7, report generation. A layered architecture is adopted, independent operation of each module of the system is ensured, and expansion and maintenance are facilitated. Data extraction, cleaning and integration processes are simplified by introducing an automatic tool, and redundant calculation is reduced by optimizing SQL and solidifying logic. Bottom logic is optimized for large-scale data volume and complex query, and report generation performance is improved. And monitoring and management functions of the whole life cycle of the data flow are provided, and data quality and report accuracy are ensured.

The invention particularly relates to a data analysis method and system. The method comprises the steps of extracting original data corresponding to a target service from a target data source according to a pre-configured data extraction strategy of the target service, loading the original data to an operation data storage layer of a data warehouse, and storing the original data; operating a data storage layer to output the original data to a detail data layer; the detailed data layer performs data preprocessing on the original data to obtain basic data and outputs the basic data to the summarized data layer; the summarizing data layer performs aggregation processing on the basic data according to a current data processing task corresponding to the target business, obtains structured data corresponding to the target business, and outputs the structured data to the application data service layer; and the application data service layer performs data analysis on the structured data according to the service configuration information corresponding to the current data processing task, and outputs a corresponding data analysis result. According to the scheme, efficient and accurate data management and analysis can be realized.

The invention provides a multi-platform collaborative data acquisition and analysis method, and relates to the technical field of data processing and analysis. The method comprises the following steps: receiving and converting data from different platforms into a unified intermediate format through a data adaptation intermediate layer, and processing a data structure and an interface of a specific platform by using a plurality of adapters; the data abstraction layer defines a standardized interface, so that multiple platforms request data through a unified interface, and a mapping rule converts the request into calling of the adaptation layer; the data integration engine preprocesses the data to ensure data consistency and accuracy; the data warehouse adopts a distributed architecture to ensure the service continuity through load balancing, data partitioning, a caching mechanism and a redundant storage and fault transfer mechanism; and the data processing and analysis module converts the data into decision suggestions by using a visual tool and a data mining algorithm. According to the method, manual intervention is reduced, the data processing speed and accuracy are improved, the data mode and trend are deeply mined, and service decision is promoted.

The invention relates to a big data-based intelligent electricity sale quantity prediction method, which comprises the following steps of S1, collecting multi-dimensional data related to electricity sale quantity, and establishing a unified data warehouse S2, constructing multi-dimensional features; s3, using a feature importance algorithm and a mutual information algorithm to quantify the relation between the features and the target variables, and rejecting redundant features; s4, constructing an intelligent electricity sale quantity prediction model based on the LSTM-TFT hybrid model; and S5, processing the real-time streaming data by using Spark Streaming, generating online features, deploying a trained model LSTM-TFT hybrid model, and obtaining a prediction result through TF-Serving. According to the method, the time sequence and the external features are dynamically modeled, so that accurate prediction of the electricity sale quantity is realized.

The invention relates to the technical field of multi-source data synchronization, in particular to a cross-library data real-time synchronization method based on a CDC technology. Data change is synchronized to a target system in real time through code-free graphical configuration, so that synchronization from a source database to a data warehouse, a data analysis platform or other business systems is realized, secondary code development is not needed, business personnel with low technical threshold and non-technical background can realize cross-database real-time synchronization of data through interface configuration, and the data synchronization efficiency is improved. Communication and cooperation among different departments are enhanced, and the working efficiency is improved.

The invention discloses an urban traffic intelligent management method based on multi-source data, and relates to the technical field of traffic data management, and the method comprises the steps: collecting multi-source traffic data, firstly carrying out the preprocessing, then carrying out the matching and integration, and building a unified traffic data warehouse; carrying out batch processing and stream processing on warehouse data by applying a distributed computing framework and a stream processing technology; on the basis of multi-source data, various traffic models are constructed and optimized, and a visual tool is developed to visually display an analysis result; by combining real-time traffic data, a model prediction result and historical data, traffic signals are dynamically adjusted, route planning and navigation services are provided for users, and routes and scheduling of public traffic are optimized; a traffic monitoring center is constructed, and real-time monitoring and timely early warning are carried out based on the traffic signal dynamic adjustment result and the real-time traffic data; establishing an emergency plan; and establishing a data sharing platform, and sharing the related traffic data. The system is used for monitoring, analyzing, managing and applying the traffic data.

The invention discloses a computer resource allocation management system and method based on data analysis, and relates to the technical field of computer resource management.The method comprises the steps that data are collected in real time, and historical data are collected; constructing a spatio-temporal data warehouse based on the real-time data and the historical data; extracting static features and dynamic features of the task, and combining the static features and the dynamic features to generate a high-dimensional task feature vector; constructing a resource semantic knowledge graph, and defining a computing resource entity, a relationship and a constraint rule; combining the spatio-temporal data warehouse and the resource semantic knowledge graph to construct a demand prediction model, and outputting probability distribution prediction of future resource demands; and in combination with real-time data, calculating resource use efficiency and a load balancing index in real time, and dynamically adjusting a resource allocation decision. The demand prediction model is constructed based on the spatio-temporal data warehouse and the resource semantic knowledge graph, resource allocation is carried out actively, resource shortage or idle is avoided, and the resource utilization rate is improved.

The invention discloses an Oracle removing method based on Clickhouse, which comprises the steps of data model design, real-time and off-line data complete storage and Clickhouse complete monitoring, and is characterized by comprising the following steps of: 1, designing a data model based on a materialized view of the Clickhouse; 2, developing offline and real-time tasks based on Spark and Flink, and respectively writing results into Hive and Topic; step 3, completely synchronizing the off-line and real-time data bins to the Clickhouse; and step 4, on the basis of the Grafana and the Prometheus, the Clickhome is monitored, and the Clickhome is monitored on the basis of the Grafana and the Prometheus. And a set of online query analysis model construction method for replacing Oracle from a data model to data storage, data analysis and Clickhome monitoring is formed.

The invention discloses a cable fault accurate positioning system based on an intelligent sensor system, and relates to the technical field of power fault positioning, and the cable fault accurate positioning system comprises a cloud monitoring center which is in communication connection with the following modules: a signal acquisition and processing module which is used for deploying the intelligent sensor system along a cable, operation environment data and traveling wave signals in the cable operation process are obtained, and the signals are preprocessed and stored in a cloud data warehouse. According to the invention, the front band-pass filter and the self-adaptive trap filter are combined to filter out high-frequency harmonic waves and specific-frequency interference signals, and the differential sampling technology is adopted to counteract common-mode interference, so that the signal-to-noise ratio of the signals is remarkably improved, a system can accurately capture and process weak traveling wave signals caused by a high-resistance grounding fault in a complex environment, and the fault detection accuracy is improved. Therefore, the fault positioning precision is greatly improved, the positioning error caused by signal interference is reduced, and the determination of the fault position is more reliable and accurate.

The invention relates to the technical field of infectious disease prevention and control, in particular to an infectious disease epidemic situation space-time aggregation analysis method and system, and the method comprises the steps: constructing a case growth trajectory propagation model and an epidemic situation propagation and diffusion model through a data warehouse; predicting an epidemic situation diffusion path and a time-space coverage change graph by using the case growth trajectory propagation model and the epidemic situation propagation and diffusion model; and fusing the predicted epidemic situation diffusion path and the time-space coverage change map with the spatial data of the key public place, and calculating to obtain an epidemic situation influence index of the key public place. By constructing the case growth trajectory propagation model and the epidemic situation propagation and diffusion model, the diffusion path and the space-time coverage change of the epidemic situation can be predicted more accurately, a more scientific basis is provided for prevention and control work, and according to the predicted epidemic situation diffusion condition and the key public occasion epidemic situation influence index, the prediction of the epidemic situation is realized. Medical resources, manpower, materials and the like can be configured more reasonably, and the prevention and control efficiency is improved.

Embodiments of the present disclosure provide a data processing method, apparatus based on data Lake and data Warehouse integration, and an electronic device. The method includes: pre-storing at least two protocol files; receiving a first metadata acquisition request; determining a target protocol file for processing the first metadata acquisition request from the at least two protocol files according to an engine type of the data processing engine which sends the first metadata acquisition request; and parsing, based on the target protocol file, the first metadata acquisition request and acquiring metadata corresponding to the first metadata acquisition request from the metadata storage space. Therefore, for at least two sets of external protocols, one set of metadata is stored, accordingly, the data storage capacity can be reduced; with the setting of one set of metadata, the metadata accessing rights management can be unified, and the security of data can be improved.

The invention relates to a query method based on query auxiliary information. The method comprises the following steps: acquiring a natural query statement of a user and user information; obtaining query auxiliary information according to the user information; determining candidate query fields associated with the natural query statement by matching query fields in a preset constellation data model; the natural query statement, the candidate query fields, the query auxiliary information and a preset first task processing text are input into a first text processing model, the query intention of the natural query statement is understood through the first text processing model in combination with the query auxiliary information, and the candidate query fields conforming to the query intention are determined as target query fields. And further, determining at least one target data table only comprising the target query field from the constellation data model, and inputting the natural query statement and the target data table into a query instruction generation model to generate a query instruction. And finally, based on the query instruction, querying from the data warehouse to obtain a query result of the natural query statement.

The invention relates to the technical field of big data processing, in particular to an auditing big data information collection method, which comprises the following steps of S1, identifying and classifying various data sources; s2, making a corresponding acquisition strategy according to the features of the various data sources identified in the S1; s3, acquiring data from each data source in real time according to the acquisition strategy formulated in S2; s4, storing the data acquired in the S3 into a data warehouse according to a preset data structure; s5, performing quality evaluation on the data stored in the data warehouse; s6, information in the data warehouse is periodically updated through a data synchronization mechanism, efficient collection, standardized processing and intelligent quality control of multi-source data are achieved through a collection strategy and an innovative data quality evaluation method, and the automation degree of data processing and the reliability of auditing analysis are remarkably improved.

The invention relates to the technical field of data processing, and discloses a method, device and equipment for constructing a lightweight data warehouse based on an MPP architecture, and the method comprises the steps: collecting data in a plurality of mainstream databases through a DataX collection assembly, and writing the collected data into a Dores data lake to obtain offline data; the method comprises the following steps: acquiring data of a plurality of data sources through a RoutingLoad component and a Kafka component, and writing the data into a Dores data lake to obtain real-time data; establishing a data hierarchical structure; performing calculation and analysis processing to generate a data calculation result; constructing a data asset catalog based on a metadata system of a Dores data lake to perform classified management of libraries, tables and fields; the data service interface is constructed based on the data asset directory, the BI report tool is connected with the data service interface to generate the data analysis report, and the data calculation result and the data analysis report are output to the outside, so that the construction and maintenance cost of an enterprise is reduced, and the requirements in a big data environment can be better met.

The invention relates to the technical field of medical health big data analysis, and discloses a disease trend analysis and early warning system and method based on big data, and the method comprises the steps: 1, obtaining multi-source medical data through a hospital information platform, carrying out the preprocessing of the data, building a correlation mapping relation, and forming a standardized data warehouse; 2, constructing a spatio-temporal data cube, and forming a three-dimensional relation graph of time, space and diseases; 3, establishing a disease analysis model according to the multiple influence factors, and performing dynamic analysis on the disease trend and the high-incidence diseases based on the disease analysis model; and 4, according to an analysis result, identifying a disease critical area, generating area early warning information and feeding back the area early warning information to a corresponding terminal. According to the method, the disease monitoring accuracy and real-time performance are effectively improved, the early warning capability is improved, resource configuration is fully integrated, and the disease response efficiency is improved.

The invention relates to the technical field of natural language processing, in particular to a method and system for achieving intelligent number asking based on a large model, and the method comprises the following steps: integrating data from various data sources including a database, a data warehouse, WEB service and Excel files; the data is cleaned, converted, standardized and normalized through the data management center, so that the data quality is ensured, and basic data support is provided for subsequent intelligent question analysis; the beneficial effects are that the method for realizing intelligent number asking based on the large model provided by the invention responds to complex and changeable data requirements in real time, helps an enterprise to quickly make a data-based decision, and reduces the time cost of waiting for the assistance of an IT department or a data analysis team; through effective integration and intelligent search of mass data resources, various data in an enterprise can be fully mined and utilized, and idleness and waste of the data resources are avoided.

The invention discloses a water conservancy data analysis method and device based on a self-attention mechanism and a medium, and relates to the technical field of water conservancy. The method comprises the following steps: constructing a water conservancy time sequence data warehouse, and establishing a water conservancy source data access mechanism to access to-be-processed water conservancy time sequence data; performing standardized preprocessing on the to-be-processed water conservancy time sequence data to obtain standard time sequence data; inputting the standard time sequence data into a pre-trained water conservancy data analysis model, and calculating attention weight distribution between different time steps and different variables through a self-attention mechanism unit in the water conservancy data analysis model; according to the attention weight distribution, performing weighted summation on the standard time sequence data, generating a spatial-temporal feature representation vector, inputting the spatial-temporal feature representation vector into a gating circulation unit in the water conservancy data analysis model, and executing spatial-temporal information fusion by updating a gate and resetting a gate; and performing water conservancy data trend prediction based on the fused features, and outputting flood prevention scheduling decision support data.

The invention relates to a travel destination recommendation method and system based on big data and computer equipment, and the method comprises the steps: collecting the original data of travel history, browsing history and social media activities of a user from multiple channels, and constructing a user behavior data warehouse after preprocessing; key features reflecting user travel modes and preferences are extracted from a user behavior data warehouse, user feature vectors are constructed according to the key features, and a candidate destination recommendation list is generated through a mixed recommendation model. And constructing a feature vector according to geographic positions, climate conditions and resource endowment data of the candidate destinations, obtaining an optimal access time period through an ecological influence evaluation model, and matching with a recommendation list to screen out a proper destination. And finally, in combination with multiple preferences of the user, a complete travel plan is generated by using a reinforcement learning algorithm, and the purposes of accurate recommendation, ecological protection and real-time dynamic adjustment are achieved.