134 results about "Dimension table" patented technology

The invention discloses a high-performance AIOT real-time data processing method and device based on Flink, Kafka and Dores, and the method comprises the steps: constructing an end-to-end hierarchical architecture, capturing multi-source database change data in real time through Flink CDC, and analyzing semi-structured data in combination with Apache NiFi, and generating a standardized data stream; the method comprises the following steps: realizing message routing based on KafkaTopic division of a service domain, ensuring the consistency of a data structure by using Schema Registry, and optimizing the transmission efficiency through a dynamic compression algorithm; an Flink real-time computing engine is adopted to implement window aggregation, asynchronous dimension table association and incremental state persistence strategies, and the resource fluctuation and state management problems in streaming computing are solved; time pre-partitioning and intelligent cold and hot layering are achieved through a Dores storage layer, the query performance is improved in combination with materialized views and multi-level indexes, and finally a full-link closed loop of collection-transmission-calculation-storage-management is formed. According to the invention, the problems that the real-time performance and the consistency are difficult to balance and the system complexity is high in the prior art are solved.

The invention discloses a user-defined multi-dimensional analysis configuration method, system and equipment based on a constellation model and a medium, belongs to the technical field of data analysis, and aims to solve the technical problem that basic data such as dimensions, indexes and data models in multi-dimensional analysis are complicated to construct. According to the technical scheme, according to the method, the dimensionality, the index and the incidence relation between the fact table and the dimensionality table configured by the model table are imported, the mapping between the dimensionality tables is automatically imported into the correlation table, the attribute and the model relation, the model maintenance cost is reduced, and the model maintenance time is shortened; ordered model construction is realized through dimensions, indexes, fact tables, dimension tables, data models and self-defined multi-dimensional query, multi-fact-table dimensions screen application configuration scenes of reverse mapping of property of the fact tables, required indexes are flexibly selected from the fact tables through dragging operation, and according to a cascade mode of the indexes and the dimensions, the application configuration scenes of reverse mapping of property of the property of the fact tables are selected to be matched with the property of the property of the property of the property of the property of the property of the property of the property of the property of the property of the property of the property. And automatically screening out the generality dimension for display.

The invention discloses a role playing dimension implementation method and system, and relates to the technical field of data processing. Comprising the steps that a meta-model of a dimension model related to a role playing dimension is created, the meta-model comprises five types of data tables, a role playing dimension name is selected from a role playing dimension name source according to requirements, when the role playing dimension name is different from a physical dimension table name, it is considered that the dimension corresponding to the role playing dimension name is the role playing dimension, and the role playing dimension is selected from a physical dimension table name; when the related dimension of one index is displayed, the role playing dimension name is used for replacing the physical dimension name to display the actual business meaning of the dimension, when the dimension is selected to summarize the indexes, the role playing dimension name is used for constructing a dynamic SQL, and when the composite index is analyzed, the actual business meaning of the dimension is displayed. When multiple indexes are analyzed in parallel, if the facts where the polyatomic indexes forming the composite indexes are located are all associated with the same physical dimension, the composite indexes are summarized through the physical dimension, and when the multiple indexes are analyzed in parallel, if the facts where all the indexes participating in analysis are located are all associated with the same physical dimension, the multiple indexes are summarized through the physical dimension; and the associated dimension list is displayed by using the physical dimension name, so that confusion caused by different role names is avoided.

The embodiment of the invention provides a data acquisition method and device, electronic equipment and a storage medium, and belongs to the technical field of data processing. The method comprises the following steps: receiving a business data query request, wherein the business data query request comprises a screening condition and a target business name of a target business; matching the target service name with service names in a pre-configured dimension table to obtain a matching result, the dimension table comprising a plurality of service names and a service code corresponding to each service name; and under the condition that the matching result indicates that the target service name corresponds to the first service code, acquiring data related to the target service from a database according to the screening condition and the first service code. According to the embodiment of the invention, the service data acquisition efficiency can be effectively improved through the pre-configured dimension table, and the query difficulty is reduced.

The invention discloses a meteorological data processing and storage method and system, and relates to the technical field of intelligent meteorological data processing, and the method comprises the steps: collecting multi-source meteorological observation data, radar data and numerical forecasting data, carrying out the access judgment, obtaining an access data set, carrying out the standardization and elevation correction, and obtaining a data set; calculating a comprehensive quality score to obtain a quality mark data set; screening samples based on the mass label data set to obtain an alignment data set, obtaining grid fusion data through the alignment data set, and performing smooth processing to obtain a fusion data set; obtaining a sealing partition based on the fused data set, and revising by using a late sample to obtain a versioned sealing partition; and obtaining a fact table and a dimension table according to the versioned sealing partition, establishing a topic domain model and a multi-version query interface, and calculating a prediction deviation and a hit rate. According to the method, unit standardization, spatial elevation correction and comprehensive quality scoring are performed on the accessed data set, so that dynamic evaluation and reliable marking of the meteorological data quality are realized.

The invention discloses a data query method, an acceleration device, computing equipment and a storage medium, and relates to the technical field of databases. The method is applied to an acceleration device of computing equipment and comprises the steps that vectorization execution is conducted on a data query request, a vector dictionary and a vector index are generated based on a fact table, a dimension table and a connection key indicated by the data query request, and multi-table connection query is achieved. On the other hand, the whole process is executed by the hardware acceleration device, the computing power of a processor in the computing equipment is unloaded, and the overall query performance can be improved.

The embodiment of the invention provides a data processing method and device, electronic equipment and a computer program product, and relates to the technical field of big data. The method comprises the steps that to-be-processed stream data is acquired, a service database table corresponding to the to-be-processed stream data is determined, and the service database table is used for generating an association dimension table; subscribing the incremental data broadcast message according to the data mapping information between the service database table and the message queue theme; in response to the received incremental data broadcast message, determining service database table incremental data corresponding to the service database table, and synchronizing the service database table incremental data to a local cache of the dimension table association logic; and executing the dimension table association task based on the local cache of the dimension table association logic to obtain a real-time association dimension table. According to the method and the device, dimension table association is carried out through data broadcasting and local cache, millisecond-level updating of data can be realized on the premise of ensuring task performance, the purpose of real-time data updating of services is achieved, and stability and consistency of dimension table data are ensured.

The invention provides a Text-to-SQL (Structured Query Language) conversion method based on semantic modeling, which relates to the technical field of natural language processing and data analysis, and comprises the following steps: designing a structured semantic model comprising a fact table and a dimension table by adopting a dimension modeling method, and defining business field division, data model definition and model dictionary construction; performing word segmentation processing on the natural language question input by the user, matching the natural language question with dimensions, indexes and dimension values in the structured semantic model based on the model dictionary library, and generating Schema information containing database types, table names, field definitions and filtering conditions; calling the large model, and converting the natural language problem into a preliminary semantic SQL in combination with Schema information and a preset rule; calling the large model again, performing logic verification and grammar optimization on the preliminary semantic SQL, and generating a corrected semantic SQL; based on Schema information and data model definition, the corrected semantic SQL is converted into an executable database SQL statement.

The invention discloses an enterprise-level index management system, method and device based on big data and a medium, and the system comprises a system operation interface which is used for obtaining table configuration; the data acquisition interface is used for acquiring initial data; the data table management platform is used for constructing a business theme table and a dimension table according to the business original table based on table configuration and initial data and determining an association relationship; the index management platform is used for creating business indexes according to the business theme table and the dimension table, constructing a corresponding first structured query statement according to the association relationship, determining a target index to be subjected to association analysis from the business indexes, and constructing a corresponding second structured query statement based on the target index; and the system operation interface is also used for constructing a visual graph for display according to the business indexes and a preset analysis model. Through centralized environment definition, storage and management of business indexes, the uniformity of index definition is ensured, the conversion process of index data can be recorded and displayed, and troubleshooting and compliance auditing are facilitated.

The invention relates to the technical field of electric digital data processing, in particular to a user question-based dimension model wide table generation method, which comprises the following steps of: converting a natural language question input by a user into a standardized question text; performing semantic analysis on the standardized question text, and extracting key elements of the dimension model; determining at least one target dimension and a hierarchical structure corresponding to the target dimension in a pre-constructed dimension knowledge graph based on the key elements; based on the target dimension and the hierarchical structure, constructing a dimension model comprising a fact table and a dimension table; and generating a physical wide table based on the dimension model. According to the scheme, through deep fusion of natural language processing and the knowledge graph, the questions of the user can be mapped to the dimension knowledge graph after semantic analysis, dimension extraction and hierarchy generation are automatically completed, and the manual modeling cost is reduced.

The invention discloses an unmanned vending machine point location alarm method and system based on Flink + Dores multi-dimensional statistical analysis, and the method comprises the steps: collecting external data streams of equipment transaction, sensors, weather / holidays and the like in real time through Flink, generating minute-level statistical indexes through window calculation, and associating a dimension table in Dores to construct a multi-dimensional real-time data stream; calculating a dynamic threshold value by adopting a space-time dynamic weight algorithm in combination with historical data pre-aggregated by the Dores star model; multi-dimensional alarm conditions are defined through an SQL-like rule engine, composite alarm triggering of cross-time periods, geographic positions and equipment states is supported, and work orders are automatically distributed to events which are abnormal and last for more than 30 minutes in combination with a network quality perception suppression false alarm and hierarchical upgrading mechanism. The method solves the problems that a traditional scheme is insufficient in real-time performance, single in dimension and rigid in static threshold value.

The invention relates to a metadata-driven data consanguinity analysis and influence evaluation method, system and device and a medium. The method comprises the following steps: performing parameter analysis on an original query language script, and constructing a parameter function model in combination with a parameter dimension table; performing parameter replacement on an original query language script according to the model, and analyzing the replaced script to obtain a table-level blood relationship, a field-level blood relationship and the like; constructing a record-level query template based on the field-level blood relationship and the conversion logic, and constructing a dynamic blood relationship map; and executing a record level query template, carrying out record node mounting on the atlas, obtaining and responding to a data change event, carrying out hierarchical influence propagation calculation, obtaining and responding to a data exception event, carrying out root cause node positioning based on the mounted dynamic blood relationship atlas, and generating a repair suggestion. According to the method, through dynamic parameter analysis and record-level blood relationship analysis, the accuracy and automation level of data blood relationship analysis are improved, and the efficiency of data governance is improved.

A method and one or more non-transitory storage media for materialized view-based query rewrite are provided. A plurality of materialized views is created based on a fact table and a dimension table. Each materialized view is based on a join between the dimension table and the fact table based on a respective foreign key column of the fact table. A database management system executes a query against the fact table and the dimension table, the query requiring one or more joins between the dimension table and the fact table based on one or more foreign key columns. For each given join of the one or more joins, responsive to the given join satisfying one or more rewrite criteria, the query is rewritten to replace the join between the dimension table and the fact table with a join between a respective materialized view for the given join and the fact table.

The invention discloses an equipment data unified integration and supply method based on dynamic metadata and intelligent routing and an Internet of Things home platform, and the method comprises the steps: obtaining to-be-updated equipment data, classifying the equipment data according to the data structure, data increment and change frequency of the equipment data, and respectively marking the equipment data as main data or transaction data; based on task characteristics of the main data and the transaction data, intelligent scheduling is carried out by utilizing dynamic node pressure sensing, and different types of data are distributed to optimally matched processing nodes; and performing corresponding update processing on the main data and the transaction data, uniformly storing update processing results in a dimension table mode and a fact table mode, and enabling the update processing results to be called by a smart home language model through a uniform data service interface.

The application discloses a meteorological data processing and storage method and system, and relates to the technical field of intelligent meteorological data processing. The method comprises the following steps: collecting multi-source meteorological observation data, radar data and numerical prediction data, performing access determination to obtain access data sets, performing standardization and elevation correction on the access data sets, calculating comprehensive quality scores to obtain quality marked data sets; filtering samples based on the quality marked data sets to obtain aligned data sets, obtaining grid fusion data through the aligned data sets and performing smoothing processing to obtain a fusion data set; obtaining a sealing partition based on the fusion data set, and revising the sealing partition by using a late sample to obtain a versioned sealing partition; obtaining a fact table and a dimension table according to the versioned sealing partition, establishing a subject domain model and a multi-version query interface, and calculating a prediction deviation and a hit rate. Through unit standardization, spatial elevation correction and comprehensive quality scoring on the access data sets, dynamic evaluation and reliable marking of meteorological data quality are realized.

In various examples, table classification based query join reordering for relational database systems and applications are provided. In some embodiments, a relational database system is provided that includes a join optimizer that evaluates a join clause of a query and categorizes relational database tables as either fact tables or dimension tables based on a normalized cardinality statistic. The join optimizer uses the fact and dimension tables to deconstruct the query into a plurality of deconstructed query join trees. Individual deconstructed query join trees may be generated for each respective fact table. The deconstructed query join trees may be joined to generate a reordered join solution representing a sequential join of the plurality of deconstructed query join trees. An updated query may be generated based on the reordered join solution, and a query response generated that answers the query based at least on the updated query.

The invention discloses a data processing method, device and equipment, a computer medium and a product, and is applied to the field of data processing.The method comprises the steps that in response to detected business operation of data change of an upstream business system, business data are collected from a database log; dividing the business data into real-time business data and dimension table business data based on the updating frequency of the business data; sending the real-time business data and the dimension table business data to a streaming computing platform through a database acquisition component; storing the dimension table service data in a preset initial increment dimension table cache through the streaming computing platform to obtain an updated increment dimension table cache; on the basis of the service identification information, associating the real-time service data with dimension table service data in the update increment dimension table cache to obtain service result data; and the streaming computing platform sends the service result data to a data subscription demander through the message middleware. According to the method, data processing efficiency is improved, and computing resources are saved.

This invention discloses a method and system for implementing role-playing dimensions, relating to the field of data processing technology. It includes: creating a meta-model of dimension models related to role-playing dimensions, the meta-model comprising five types of data tables; selecting role-playing dimension names from their sources according to requirements; when a role-playing dimension name differs from a physical dimension table name, the dimension corresponding to the role-playing dimension name is considered a role-playing dimension; when displaying the related dimensions of an indicator, using the role-playing dimension name to replace the physical dimension name to demonstrate the actual business meaning of the dimension; when summarizing the indicator using selected dimensions, constructing dynamic SQL using the role-playing dimension name; when analyzing composite indicators, if the facts of the multi-atomic indicators constituting the composite indicator are all associated with the same physical dimension, then the composite indicator is summarized using that physical dimension; when performing parallel analysis of multiple indicators, if the facts of each indicator involved in the analysis are all associated with the same physical dimension, then the multiple indicators are summarized using the physical dimension; and displaying the list of related dimensions using the physical dimension name to avoid confusion caused by different role names.

The invention discloses a data warehouse model design method, and belongs to the technical field of data warehouse modeling. According to the method, a dimension modeling method is adopted to construct and analyze a mart model, data is organized into a non-standardized relation model, and the relation between tables is defined through keywords and foreign keys. And designing a dimension table system comprising a product dimension, a calendar day dimension, an institution dimension, an insurance policy attribute dimension and a customer dimension, and a fact table system comprising an insurance claim settlement analysis fact table, a payment analysis fact table, a claim settlement efficiency analysis fact table and a corresponding intermediate fact table. The analysis subject domain data is formatted into a star model or snowflake model, where the analysis fact table is a lightly summarized transactional fact table, and the intermediate fact table is a versioned historical table. According to the method, repeated data construction is avoided through a unified modeling method, and the cost is reduced; an efficient query analysis function is provided by optimizing a model structure; and the data traceability is ensured through versioning management.

This application provides a method, device, and storage medium for dimension table joins in Flink, relating to the field of big data technology. The method identifies a target subtask among multiple subtasks of a unified dimension table join operator in the current task manager. After controlling the target subtask to send a query request to an external database and obtain the dimension table data sent by the external database, the method controls the target subtask to load the dimension table data into a static dimension table object set in the metadata space of the task manager. This avoids putting significant operational pressure on the external database in a short period. The method controls the target subtask and multiple other subtasks to perform dimension table join operations based on the obtained raw data and the dimension table data in the static dimension table object, allowing all subtasks in the same task manager to share the same dimension table data, thereby saving physical memory resources.

The invention provides a data processing method and device, electronic equipment and a readable storage medium, the data processing method is used for a server, and the data processing method comprises the following steps: receiving update data and an update data request sent by a client; acquiring historical data and an initial dimension table of the historical data according to the data updating request; acquiring updated data and a first dimension table of the updated data according to the updated data, the initial dimension table and the historical data; responding to a query request sent by a client, and extracting query data information in the query request; and obtaining query data based on the updated data, the first dimension table and the query data information, and transmitting the query data to the client. During multi-level data query, the level dimension table is introduced, and data information needing to be queried can be searched in stored data by referring to the level dimension table.

The present application provides a method, a computing device, and an electronic device for computing stream data. The method includes: obtaining the primary key value corresponding to the target stream data and the primary key value corresponding to the dimension table, and obtaining the target stream data primary key value and the dimension table primary key value; determining the first target partition of the dimension table according to the partitioning strategy based on the dimension table primary key value, and obtaining the first mapping relationship between the first target partition and the node, determining the target node corresponding to the first target partition according to the first mapping relationship, and storing the dimension table to the target node; determining the second target partition of the target stream data according to the partitioning strategy based on the target stream data primary key value, and obtaining the second mapping relationship between the second target partition and the node, determining the target node corresponding to the second target partition according to the second mapping relationship, and storing the target stream data to the target node, so as to perform streaming computing on the target stream data and the dimension table on the target node. This solves the problem of decreased stream data computing performance caused by storing a large number of dimension tables on the same node.

The embodiment of the application provides a data query method, system, device, equipment and storage medium, and relates to the technical field of data processing. The specific implementation scheme is: receiving a first statistical query statement, determining a specified statistical range, a specified statistical method and a specified measurement indicated by the first statistical query statement; if there is no aggregate table containing to-be-queried data in each aggregate table stored in advance, determining whether there is a target aggregate table in each aggregate table, the statistical range of which belongs to the specified statistical range, the statistical measurement of which is the specified measurement, and the statistical method of which is the specified statistical method based on the hierarchical structure between each dimension table stored in advance; if there is, obtaining the measurement value of the specified measurement corresponding to each statistical range belonging to the specified statistical range from the target aggregate table based on the hierarchical structure, and obtaining a query result based on the obtained measurement value and the specified statistical method. It can be seen that, by the scheme, the query efficiency can be improved.

The present invention relates to the technical field of multidimensional data analysis in big data, and discloses an intelligent dynamic dimensional sorting method and system based on multidimensional feature parameters, the method comprising: creating a commodity hierarchical dimension table containing the atomic dimensions and hierarchical relationships of commodities according to the standard dimensional data of commodities stored in a data warehouse; creating a commodity parameter dimension table containing all parameter types and corresponding parameter values ​​of commodities according to the parameter types corresponding to each commodity and in combination with the hierarchical dimension table; disassembling commodities of different levels according to parameter types, and performing dimensionally differentiated sorting on each parameter value after disassembly through a sorting algorithm. The intelligent dynamic dimensional sorting method and system based on multidimensional feature parameters provided by the present invention solve the problem that traditional business multidimensional definitions cannot be controlled, reduce physical storage space and manual management costs through intelligent analysis and sorting, and can help business data analysts gain insight into key issues more quickly.

The invention discloses an Internet of Things data storage method and system based on data warehouse modeling, and belongs to the field of Internet of Things data storage systems, and the method comprises the steps: obtaining a business data source table, extracting multi-valued attribute data and slow change dimension data, and constructing a first dimension table and a second dimension table; storing the multi-valued attribute data into one attribute field of the first dimension table, storing the multi-valued attribute data into a plurality of attribute fields of the first dimension table, or storing each dimension value in the multi-valued attribute data into a single row of the first dimension table; and writing the slow change dimension data into the second dimension table in a rewriting manner, writing the slow change dimension data into the second dimension table in a manner of adding new rows, writing the slow change dimension data into the second dimension table in a manner of adding new attribute fields, or writing the slow change dimension data into a micro dimension table associated with the second dimension table. According to the method, multiple processing methods of the data warehouse model on the multi-valued attribute data and the slowly-changing dimension data are supported, and flexible processing of the data is achieved in multiple storage modes.

The invention discloses a data warehouse construction and application method based on multi-source data fusion, and relates to the technical field of data fusion, and the method comprises the steps: multi-source data integration layer construction: obtaining structured, unstructured and semi-structured data through a heterogeneous data collection module, carrying out the automatic preprocessing and standardization processing of the data, and obtaining a multi-source data integration layer; the standardization processing comprises a unified naming rule, a coding rule and a data type; designing a core model of the data warehouse: constructing a star model based on a business theme, the star model comprising at least one fact table and a plurality of associated dimension tables, the fact table storing business index data, and the dimension tables storing descriptive attribute data; implementing an ETL automatic process: configuring a data extraction and conversion rule and a scheduling strategy through a visual ETL tool, automatically loading the data processed by the integration layer to a star model, and monitoring a task state in real time; and intelligent query and application recommendation.

The invention relates to a general decision log real-time library falling method based on flink, and the method comprises the steps: firstly, writing a newly-added log of a risk control decision system into kafka in real time by using an OGG tool, merging a decision configuration input parameter field and a non-decision input parameter field, synchronizing the merged fields to a target configuration table, and then writing the merged fields into a kafka specific topic; then, target configuration table data are spliced into a long character string by using flink, and the long character string is written into an oracle dimension table; and finally, the flink program associates the kafka log data with the oracle dimension table in real time, dynamically analyzes key information, generates a json field and writes the json field into a Dores analysis library, so that real-time and efficient library falling of the log is realized, and the problems of log content redundancy, low development efficiency and poor data timeliness in the prior art are solved.

The invention provides a dimension data model quality evaluation method and device, a storage medium and a program product, and relates to the technical field of databases, the method comprises the following steps: firstly, based on metadata of a plurality of data tables, determining a dimension table and a fact table of a dimension data model; then, according to the dimension table and the fact table, determining a graph structure model corresponding to the dimension data model; and performing quality evaluation on the dimension data model according to the graph structure model. In the embodiment of the invention, the graph structure model corresponding to the dimension data model is obtained, and the quality of the dimension data model is evaluated based on the graph structure model, so that automatic evaluation of the quality of the dimension data model can be realized, and the quality evaluation efficiency, flexibility and accuracy of the dimension data model are improved.

The invention discloses a method and a device for generating a commodity pool based on a new machine recommendation algorithm of big data. The method comprises the following steps: constructing a basic view through a Spark framework; performing popularity quantitative evaluation through a commodity popularity scoring model, and respectively screening out hot-sell commodities and new commodities; performing data matching through an SQL linked table query operation, and generating an initial candidate commodity set and a quantity target value of each type of commodities; calculating to obtain a comprehensive score through a multi-objective optimization model; partition-level commodity filtering and optimization are carried out through a MapReduce programming model, and an optimized commodity candidate pool is obtained; performing commodity compliance verification according to items of the white list, and based on a forbidden list in the dimension list black list, adding just-needed categories according to scene requirements through reverse connection operation to obtain a final commodity candidate pool; and merging the commodity data in the final commodity candidate pool into a hive table for persistent storage. The problems that in the prior art, consumption behavior recognition is lacked, and the requirement of a special scene cannot be met are solved.

The invention provides a three-phase asynchronous motor database construction method and system, and the method comprises the steps: obtaining the source data of a three-phase asynchronous motor, and the source data comprises the common parameters, geometric dimensions and performance parameters of the motor; analyzing a data dependency relationship among the common parameters, the geometric dimensions and the performance parameters to determine a determination condition of each data set; creating a plurality of correlative data tables based on the data dependency relationship, wherein the data tables comprise a geometric dimension table group for storing geometric dimensions of the motor and a performance parameter table for storing performance parameters of the motor; and constructing an entity-relationship model of the database based on the association relationship among the plurality of data tables, and generating an SQL statement for data query. According to the technical scheme provided by the invention, structured and standardized storage of the motor data is realized, data redundancy is remarkably reduced, data reusability and query efficiency are improved, and industrial digital application is effectively supported.