62 results about "Data mart" patented technology

A data processing and analysis system that optimizes the resources to be used for data storage and refresh events. A partitioner module for a data analysis system can receive a first client criteria and a first client dataset that includes tabular data and calculate scores that are used to generate partitioning strategies. The selected partitioning strategy can be implemented to produce aggregated data that can be stored in an intelligent data mart. The partitions can then be accessed by a data visualization platform for intelligent, dynamic responses to user requests for data analyses and generation of visualizations. By providing synchronous partitioning of data (especially big data) and intelligent refresh, the data can move from the back-end to the front-end with minimal user clicks and minimal latency in performance.

The invention discloses an AB experiment generation and optimization method based on an AI model and a local vector database. The AB experiment generation and optimization method comprises the following steps: constructing a local experiment database comprising a business knowledge base, a data mart and historical experiment data; carrying out vectorization processing on the business knowledge base, the data marts and the historical experimental data, and storing generated vectors in a local vector database; a large-scale language AI model is utilized to carry out a prompt project, full-process automation of AB experiments is realized, and the full-process automation comprises automatic experiment creation, automatic effect estimation and automatic result analysis. According to the method, the experimental design scheme can be automatically analyzed and generated through training and fine tuning by utilizing the large language AI model, and the fine tuning process enables the large language AI model to adapt to a specific business scene and accurately identify key factors influencing user behaviors, so that intelligent suggestions are provided for experimental design.

The invention discloses an enterprise information visual management system and method based on data sharing, and belongs to the technical field of enterprise information management.The method comprises the steps that by connecting business systems in an enterprise, user behavior data and business data are collected and preprocessed, and an internal data set is constructed; acquiring data from an external data source, and constructing an external data set through semantic analysis, screening and classification; internal and external data are integrated, a dynamic cleaning rule is formulated, multi-source fusion is realized through data association, data are stored in a classified manner according to business requirements, and a data warehouse with a layered architecture is constructed; meanwhile, a personalized data mart is generated based on user roles and permissions, user demands are automatically predicted, and analysis indexes and dimensions are pre-generated; and finally, constructing a visual analysis platform, generating a preliminary recommendation scheme in combination with the pre-generated content and the user operation track, fusing user-defined setting, and outputting a final visual result in real time, thereby realizing efficient management and visual presentation of enterprise information.

The invention relates to the technical field of data processing and the field of financial science and technology, and discloses a data processing method for a data warehouse, which comprises the following steps of: acquiring business data from a business database, and synchronizing the business data to an operation data storage layer of a data lake processing framework; a distributed real-time data processing module is used for processing the business data in the operation data storage layer, the processed business data are incrementally written into the operation data storage layer in a transaction control mode, incremental data are read from the operation data storage layer and written into a data warehouse layer of the data lake processing framework, and the data warehouse layer is used for storing the incremental data. The incremental data is further transmitted to the data mart layer. And performing aggregation processing on the incremental data based on business requirements, storing the aggregated data in an aggregation index output library, and finally transmitting the aggregated data to an application platform database for business query and report generation. By adopting incremental data processing and transaction control, the problems of data processing delay and data consistency are solved, and efficient real-time data processing is realized.

Adaptable data warehousing computing systems and methods are disclosed for improved and flexible declarative control for programmatic modeling and database building, including in compute intense environments. A declarative scripting engine executes computing commands defined in a domain specific language (DSL), which is configured to programmatically configure a conceptual data model defining business object(s). A logical data model may be generated based on a declarative command defining the conceptual data model. A physical data model may be generated based on the logical data model, where the physical data model defines stage, core layer, and / or a data mart layers that may be deployed to a target database or data warehouse, and where the stage, core layer, and / or a data mart layers are the same or similar to those of the logical data model.

The invention discloses a power distribution network line loss intelligent sensing and tracing decision-making method and system, and belongs to the technical field of power distribution network operation and management.The power distribution network line loss intelligent sensing and tracing decision-making method comprises the steps that multi-source heterogeneous data are connected, timestamp interpolation is carried out, and data are obtained; associating the position and GIS coordinates of the electrical equipment in the electrical network through the unique identifier of the electrical equipment, and outputting a standardized data mart; a dynamic line loss separation engine calculates total loss according to the standardized data mart and a photovoltaic back-feeding compensation item, dynamically corrects technical line loss through a photovoltaic output coefficient, subdivides a line loss component by using a K-means algorithm, and outputs a line loss component matrix; and calculating a feature vector and inputting the feature vector into the traceability decision tree to trace the line loss abnormal root cause. Time sequence and space dislocation of multi-source heterogeneous data is eliminated through timestamp intelligent interpolation and equipment ID two-dimensional mapping fusion data, accurate decoupling of management line loss and technical line loss is achieved through a corrected dynamic line loss separation engine, and the misjudgment rate of a new energy scene is reduced through a traceability decision tree.

A device and method are described for a universal analytical data mart and data structure for same. The analytical data mart (ADM) associated data structure is designed to allow data from disparate sources to be integrated, enabling streamlined business intelligence, reporting and ad hoc analysis. Conceptually, the ADM enables analytics and business intelligence from multiple frames of reference including people, such as parties and actors including individuals and organizations, places, such as addresses with geographic information at various levels of view, objects, such as insured properties, automobiles and machinery, and events, milestones which happen at points in time and provide analytical / business value.

The invention discloses a fusion method and device based on multi-source water conservancy data, equipment and a medium, and relates to the technical field of water conservancy data processing, and the technical scheme is characterized in that the method comprises the steps: collecting water conservancy original data, carrying out the classified storage of the water conservancy original data according to a water conservancy data specification, and obtaining a water conservancy classification data set; preprocessing the water conservancy classification data set to obtain a preprocessed water conservancy classification data set, combining similar data of multiple sources, and carrying out missing compensation processing on the similar data to obtain a fused similar data set; and carrying out hierarchical modeling on the fused data sets of the same kind by utilizing distributed processing and a multi-bin hierarchical architecture design to obtain a standardized unified water conservancy data mart. The system has the advantages of high-efficiency water conservancy data acquisition, high-quality similar data fusion, high-standard heterogeneous data fusion, high-availability water conservancy data service and the like.

The invention provides a data processing method and system, and the method comprises the steps: obtaining multi-modal data according to business demand data for each business, converting the multi-modal data into structured data through a specific rule, solving a management problem of unstructured data, further constructing a label system comprising static and dynamic labels, and improving the management efficiency. Multi-dimensional high-precision and high-efficiency processing of data is realized by establishing a data mart and combining generative artificial intelligence with a data bin covering granularity, dimensionality, fact tables and themes, and finally, accurate user group customized recommendation is realized by adopting customized data recommendation rules for different target crowds by constructing the data mart and combining with the generative artificial intelligence.

The invention discloses a hydropower station operation data intelligent management system and method based on a data mart, and relates to the technical field of operation abnormity identification of a hydroelectric generating set, and the method comprises the steps: dividing a working condition interval of the hydroelectric generating set; obtaining a reference feature group of all the working condition intervals; actual electric power sampling is carried out on a certain working condition interval, a curve is fitted according to sampling data points, and a first fitted curve of the working condition interval is determined through a correlation calculation method; obtaining a first fitting curve of all the working condition intervals; storing the reference feature group and the first fitting curve into a data mart; carrying out real-time monitoring on a certain hydroelectric generating set, and carrying out sampling inspection to construct a first monitoring sequence; and based on the first monitoring sequence, retrieving the data mart, obtaining a reference feature group and a first fitting curve under the same working condition, and performing early warning according to different conditions. The method has good expandability and adaptability, and is suitable for intelligent monitoring and health diagnosis in a multi-unit cooperative operation scene.

The invention provides a bank customer portrait determination method and device, and the method comprises the steps: carrying out the processing of obtained bank internal system data and bank external system data, and generating a customer data mart; constructing a customer value index system according to the customer data marts; the customer value index system comprises customer multi-dimensional value index classes, each value index class comprises at least one value index, and the customer value index system further comprises a definition of each value index; carrying out weight analysis on the customer value index system by adopting an analytic hierarchy process, and constructing a customer evaluation model; based on customer data corresponding to a target customer group in the customer data mart and a customer evaluation model, generating a customer portrait of each target customer in the target customer group, and outputting the customer portrait in an interactive interface of the terminal equipment; the customer portrait at least indicates the index value and the overall grade of each target customer under each value index class. Therefore, a more comprehensive and accurate customer portrait of the bank customer can be generated.

The present invention discloses a method for storing and detecting multi-version metadata of a power grid data warehouse, which includes the following steps: constructing a power grid Data Vault (DV) data warehouse; constructing a metamodel for a power grid DV data warehouse environment, including a DV mode and a multidimensional (MD) mode part; performing metadata consistency detection of the DV mode; performing metadata consistency detection of the MD mode; performing consistency detection of the integrity constraints of the attribute table Attributes; performing metadata consistency detection of the DV mode and the MD mode; the present invention utilizes the metamodel of the power grid metadata repository to respectively check the multi-version metadata consistency relationship between the data warehouse area, the data mart area, and the data warehouse and the data mart. The present invention can automatically discover metadata missing, duplication and conflict at the power grid metadata level, thereby achieving the purpose of improving the quality of power grid data.

The invention relates to a financial data index testing method and device, electronic equipment and a storage medium. The financial data index test method comprises the steps of generating a first data index set according to a bank supervision meeting off-site supervision 1104 report, data indexes and a mapping relation; compiling a cross-system index cross verification scheme; performing data mart table and field correspondence, establishing an index cross-check mapping relation, and developing an index cross-check SQL script set; performing blood relationship analysis to obtain a front dependency table based on a data warehouse, and desensitizing batch data of the front dependency table from production recovery to a test environment; running a data mart script scheduling job, and processing and loading the data of the data warehouse into a data mart target table; and carrying out index cross-check rule entry, generating an index cross-check rule, and generating a data index cross-check test case. According to the financial data index test method, the supervision submitted data index can be subjected to cross-system automatic verification, and the test efficiency is improved.

The present invention belongs to the field of geological mineral resource prediction, and particularly relates to a multi-scale progressive prospecting prediction method for solid minerals based on geological big data. The present invention collects multi-scale geological, geophysical exploration data, geochemical exploration data, remote sensing and other data in the study area, and uses data mining technology and machine learning methods to achieve prospecting prediction and target area delineation. This method aims to combine three-dimensional geological models, machine learning methods, and data mining technologies to process geological big data in the prediction area, determine the target area progressively at multiple scales, and provide a prospecting prediction method for mineral resource prediction; based on prospecting case review, a prospecting knowledge map is constructed, and a prospecting process knowledge association map is drawn. According to the working scale, the geological, geophysical, geochemical and remote sensing data are divided into 1:500,000, 1:250,000, 1:50,000, 1:25,000 and 1:5,000, a three-dimensional geological model and attribute data mart are constructed, and the laws between mineralization conditions and mineralization controlling elements are analyzed to form a mineralization prediction model to achieve prospecting prediction.

This invention discloses an intelligent management system and method for hydropower station operation data based on a data mart, relating to the field of hydropower unit operation anomaly identification technology. The method includes: dividing the hydropower unit into operating condition intervals; obtaining a baseline feature set for all operating condition intervals; sampling actual power for a certain operating condition interval, fitting a curve based on the sampled data points, and determining a first fitted curve for that operating condition interval using a correlation calculation method; obtaining the first fitted curves for all operating condition intervals; storing the baseline feature set and the first fitted curve in a data mart; performing real-time monitoring of a certain hydropower unit and constructing a first monitoring sequence through random sampling; and based on the first monitoring sequence, retrieving the data mart to obtain the baseline feature set and the first fitted curve under the same operating condition, and issuing early warnings according to different situations. This invention has good scalability and adaptability, and is suitable for intelligent monitoring and health diagnosis in multi-unit collaborative operation scenarios.

The invention discloses an architecture design method of an operation management system, and relates to the technical field of enterprise operation management, and the method comprises the following specific steps: firstly, through multi-source data integration and cross-system verification, data integrity and accuracy are ensured; then ETL processing and layered modeling are carried out, DIM and DW layers are constructed, and asset information is reversely written; secondly, performing secondary data processing on a Data Mart layer, screening high-credibility data, and configuring a security and scaling mechanism; and finally, realizing visual display and linkage management, presenting data in multiple dimensions, early warning abnormity, and ensuring close linkage of an operation management platform, a data model and a mart. Comprehensive and accurate data are ensured through multi-source data integration and intelligent verification, ETL intelligent processing, bidirectional blood relationship tracing and fine-grained safety control are adopted, the intuition, accuracy and safety of operation management are improved, and efficient application of data and tight linkage of a platform are achieved.

Adaptable data warehousing computing systems and methods are disclosed for improved and flexible declarative control for programmatic modeling and database building, including in compute intense environments. A declarative scripting engine executes computing commands defined in a domain specific language (DSL), which is configured to programmatically configure a conceptual data model defining business object(s). A logical data model may be generated based on a declarative command defining the conceptual data model. A physical data model may be generated based on the logical data model, where the physical data model defines stage, core layer, and / or a data mart layers that may be deployed to a target database or data warehouse, and where the stage, core layer, and / or a data mart layers are the same or similar to those of the logical data model.

The present invention relates to the field of data transaction technology, specifically a data mart transaction platform based on electricity metering, the platform includes a data structure analysis module, an abnormal pattern positioning module, an emergency management module, a transaction pattern recognition module, a transaction flow configuration module, and a price dynamic adjustment module. In the present invention, by refining the data processing flow and enhancing transaction transparency and responsiveness, significant performance enhancement is provided, and by detailed monitoring of the seasonality and trend deviations of electricity data, the platform is allowed to identify data anomalies that do not conform to historical models in real time, and realize rapid identification and response to abnormal transactions, so that potential risk behaviors can be detected and analyzed in the early stages of formation, and by dynamically adjusting the transaction volume and path of the transaction node, the data flow is optimized to ensure that the market price of electricity data reflects the supply and demand changes in real time, improve the market's operating efficiency, enhance the market's adaptability to emergencies, and improve the rationality and economic benefits of electricity distribution.

The invention discloses an industrial data exception attribution method, device and equipment, a storage medium and a product, is applied to a smart power plant data platform, and relates to the technical field of exception detection. The method comprises the following steps: acquiring multi-source industrial data in real time; performing standardization processing on the multi-source industrial data to generate a standardized data stream in a unified format; performing anomaly detection on the standardized data through a pre-trained integrated learning model, and identifying an abnormal mode in the data; and in response to the detected abnormal mode, performing multi-dimensional attribution analysis on the abnormal mode based on an association rule constructed by a data topic and a data mart, and determining a root cause of the abnormality. Through progressive processing of data integration, intelligent detection and business attribution, a complete closed loop from data perception to intelligent decision is realized, and mass data is converted into high-value decision information for directly driving business optimization.

In some implementations, an analytics platform may process a data lineage configuration to identify one or more data sources storing one or more data sets associated with a data analytics use case and identify one or more data relations associated with the one or more data sets. The analytics platform may parse a source code template defining one or more functions to build a data mart and a data pipeline that enables the data analytics use case. The source code template may include one or more tokens to specify elements that define the one or more data sources, data sets, and data relations identified in the data lineage configuration. The analytics platform may automatically generate source code that is executable to build a data mart and a data pipeline to enable the data analytics use case based on the data lineage configuration and the source code template.

The invention relates to the technical field of financial systems, and particularly discloses a centralized processing method for a certificate inquiry letter, which comprises the following steps of: acquiring a paper certificate inquiry letter, and scanning the paper certificate inquiry letter into an electronic image; extracting certification inquiry home page information and certification inquiry seal page information in the electronic image; checking the formats of the information of the certifying home page and the information of the certifying seal page, the information accuracy of the information of the certifying home page and the seal integrity of the certifying seal page; printing verification and charging are carried out based on the information of the certification inquiry home page; inputting the information of the certification inquiry home page and the information of the certification inquiry seal page into a preset certification inquiry letter template to generate an electronic certification inquiry letter; the electronic confirmation request is stored in a data mart, the data mart comprises a query interface, and the query interface is used for checking the information consistency of the electronic confirmation request and the corresponding paper confirmation request; and returning the qualified electronic confirmation request to the user equipment. According to the method, the processing efficiency of the paper inquiry certificate is improved, centralized processing of the inquiry certificate is realized, and the labor cost is reduced.

This application discloses an intelligent generation method and apparatus for decision analysis reports in the insurance field. The method includes: constructing a thematic data mart containing insurance business data, and vectorizing and semantically slicing insurance field rules and indicator texts to form a knowledge unit index; responding to a report generation request, constructing a computational agent based on a matched report template; using this agent, generating analytical text by using retrieved data and knowledge units as a context-driven generative model, and synthesizing a decision analysis report; collecting interactive operation data related to the report, and adjusting the configuration parameters of the report template accordingly. This application achieves full-process automation from multi-source data integration and professional knowledge structuring to intelligent analysis, report generation, and continuous optimization, solving the problems of low efficiency, poor accuracy, and difficulty in adapting to large-scale data and complex business logic in traditional manual analysis in the insurance industry, significantly improving decision support efficiency and intelligence.

The invention discloses an operation resource pool management system and method, and relates to the technical field of resource management, and the system comprises an operation resource and cost management module which is used for generating a project approval budget according to a special operation plan, and carrying out the association with a resource pool; the rights and interests management module is used for creating rights and interests required by operation activities and associating the rights and interests with the resource pool; the point management module is used for associating points corresponding to operation activities with a resource pool to obtain a data mart when rights and interests are triggered; and the resource pool configuration module is used for associating the data mart with the right and interest use scene, and performing optimal configuration on the resource pool according to the budget consumption condition of the operation activity. Due to the fact that the rights and interests management module and the point management module are introduced, association management is conducted on the resource pool and the issuing process of the rights and interests and the points, the situation that due to the fact that traditional resource pool management and control are not strict, redisk difficulty is large is avoided, cost and fund consumption in the marketing field can be traced and monitored, and the execution efficiency of the marketing process is improved.

This application discloses a data mart management method, apparatus, electronic device, and storage medium. The method includes: receiving data to be processed from a source system; storing the data to be processed in a predetermined table structure to obtain a standard-format data table; processing the data to be processed sequentially at multiple levels according to a hierarchical order based on the standard-format data table to obtain a processing result table of the data to be processed; and sending the processing result table of the data to be processed to a downstream system. The method of this application can standardize the creation and storage format of data tables, directly process the standardized data, reduce the application and subsequent maintenance difficulty of data tables in data marts, and improve the management efficiency of data marts.

The invention discloses a data warehouse layering and domain dividing design method, and belongs to the technical field of data warehouses, and the method comprises the steps: constructing a data warehouse, and carrying out the layering of the data warehouse; extracting traffic flow related data from a source end service system in a data operation layer, and establishing a data model in a zoning manner; in the data detail layer, standardizing and cleaning the data of the data operation layer, and establishing a data model based on the processed data; preprocessing the data of the detail layer in the data intermediate layer, establishing a fact table and performing light summarization; exchanging the data to a Dores database for processing; in the data summarization layer, summarization processing is provided for the theme library based on the data of the middle layer, and business indexes are developed according to different business dimensions; in the data mart layer, the fact table and the index data are subjected to domain processing according to business requirements and are docked to an application thematic library. According to the method, the problems of high data processing delay, long period and analysis singleness are effectively solved, and data redundancy and delay are remarkably reduced.

A data management system includes a computing device. The computing device includes one or more processors to execute instructions stored in memory of the computing device. The instructions, when executed by the computer device, cause the computing device to receive an indication that multiple data elements are ready for evaluation, as well as determine whether metadata registration for the multiple data elements is valid. The instructions also cause the computing device to determine a data quality of the multiple data elements by evaluating multiple data elements based on multiple data quality dimensions including at least two of: validity, completeness, uniqueness, accuracy, consistency, and timeliness. The instructions also cause the computing device to provide an indication of the data quality of the multiple data elements via a graphical user interface (GUI) generated via a centralized data mart.

The invention provides an event prompting method and device, and the method is applied to a customer management platform, and the method comprises the steps: building a customer data mart based on the obtained bank internal system data and bank external system data; configuring a business event rule set for the customer data mart from at least one business participation object dimension; monitoring the customer data mart by using the business event rule set corresponding to each business participation object dimension, and identifying a target business event under each business participation object dimension; in response to the generation of the target business event, determining event prompt information, an information push mode and target business personnel for the target business event according to the target business event rule; and automatically distributing the event prompt information to the target business personnel through an information push channel included in the information push mode. In this way, timely and unified monitoring and targeted personalized information pushing of the business events can be achieved.

A device and method are described for a universal analytical data mart and data structure for same. The analytical data mart (ADM) associated data structure is designed to allow data from disparate sources to be integrated, enabling streamlined business intelligence, reporting and ad hoc analysis. Conceptually, the ADM enables analytics and business intelligence from multiple frames of reference including people, such as parties and actors including individuals and organizations, places, such as addresses with geographic information at various levels of view, objects, such as insured properties, automobiles and machinery, and events, milestones which happen at points in time and provide analytical / business value.

The application discloses a business index acquisition method and device based on distributed processing and a readable medium. The method comprises the following steps: collecting background transaction data from a source database to a distributed processing system as source layer data; partitioning and storing the source layer data to obtain partition storage data corresponding to different business requirements and / or functions as model layer data; processing the partition storage data corresponding to different business requirements and / or functions in the model layer data based on a distributed processing mode to obtain corresponding marketing indexes as data mart layer data; and finally outputting the data mart layer data to a downstream business system. The application can make more accurate and customer demand-conforming marketing recommendations by introducing background transaction data into a marketing scenario. By introducing a distributed processing system and designing a hierarchical structure of a database table for data storage and processing, the application solves the problems of slow calculation, poor accuracy and stability of indexes and the like in the traditional mode for large data volume.

In some implementations, an analytics platform may process a data lineage configuration to identify one or more data sources storing one or more data sets associated with a data analytics use case and identify one or more data relations associated with the one or more data sets. The analytics platform may parse a source code template defining one or more functions to build a data mart and a data pipeline that enables the data analytics use case. The source code template may include one or more tokens to specify elements that define the one or more data sources, data sets, and data relations identified in the data lineage configuration. The analytics platform may automatically generate source code that is executable to build a data mart and a data pipeline to enable the data analytics use case based on the data lineage configuration and the source code template.