71 results about "Dimension table" patented technology

The invention discloses a high-performance AIOT real-time data processing method and device based on Flink, Kafka and Dores, and the method comprises the steps: constructing an end-to-end hierarchical architecture, capturing multi-source database change data in real time through Flink CDC, and analyzing semi-structured data in combination with Apache NiFi, and generating a standardized data stream; the method comprises the following steps: realizing message routing based on KafkaTopic division of a service domain, ensuring the consistency of a data structure by using Schema Registry, and optimizing the transmission efficiency through a dynamic compression algorithm; an Flink real-time computing engine is adopted to implement window aggregation, asynchronous dimension table association and incremental state persistence strategies, and the resource fluctuation and state management problems in streaming computing are solved; time pre-partitioning and intelligent cold and hot layering are achieved through a Dores storage layer, the query performance is improved in combination with materialized views and multi-level indexes, and finally a full-link closed loop of collection-transmission-calculation-storage-management is formed. According to the invention, the problems that the real-time performance and the consistency are difficult to balance and the system complexity is high in the prior art are solved.

The invention discloses a meteorological data processing and storage method and system, and relates to the technical field of intelligent meteorological data processing, and the method comprises the steps: collecting multi-source meteorological observation data, radar data and numerical forecasting data, carrying out the access judgment, obtaining an access data set, carrying out the standardization and elevation correction, and obtaining a data set; calculating a comprehensive quality score to obtain a quality mark data set; screening samples based on the mass label data set to obtain an alignment data set, obtaining grid fusion data through the alignment data set, and performing smooth processing to obtain a fusion data set; obtaining a sealing partition based on the fused data set, and revising by using a late sample to obtain a versioned sealing partition; and obtaining a fact table and a dimension table according to the versioned sealing partition, establishing a topic domain model and a multi-version query interface, and calculating a prediction deviation and a hit rate. According to the method, unit standardization, spatial elevation correction and comprehensive quality scoring are performed on the accessed data set, so that dynamic evaluation and reliable marking of the meteorological data quality are realized.

The invention provides a Text-to-SQL (Structured Query Language) conversion method based on semantic modeling, which relates to the technical field of natural language processing and data analysis, and comprises the following steps: designing a structured semantic model comprising a fact table and a dimension table by adopting a dimension modeling method, and defining business field division, data model definition and model dictionary construction; performing word segmentation processing on the natural language question input by the user, matching the natural language question with dimensions, indexes and dimension values in the structured semantic model based on the model dictionary library, and generating Schema information containing database types, table names, field definitions and filtering conditions; calling the large model, and converting the natural language problem into a preliminary semantic SQL in combination with Schema information and a preset rule; calling the large model again, performing logic verification and grammar optimization on the preliminary semantic SQL, and generating a corrected semantic SQL; based on Schema information and data model definition, the corrected semantic SQL is converted into an executable database SQL statement.

The invention relates to the technical field of electric digital data processing, in particular to a user question-based dimension model wide table generation method, which comprises the following steps of: converting a natural language question input by a user into a standardized question text; performing semantic analysis on the standardized question text, and extracting key elements of the dimension model; determining at least one target dimension and a hierarchical structure corresponding to the target dimension in a pre-constructed dimension knowledge graph based on the key elements; based on the target dimension and the hierarchical structure, constructing a dimension model comprising a fact table and a dimension table; and generating a physical wide table based on the dimension model. According to the scheme, through deep fusion of natural language processing and the knowledge graph, the questions of the user can be mapped to the dimension knowledge graph after semantic analysis, dimension extraction and hierarchy generation are automatically completed, and the manual modeling cost is reduced.

The invention discloses an unmanned vending machine point location alarm method and system based on Flink + Dores multi-dimensional statistical analysis, and the method comprises the steps: collecting external data streams of equipment transaction, sensors, weather / holidays and the like in real time through Flink, generating minute-level statistical indexes through window calculation, and associating a dimension table in Dores to construct a multi-dimensional real-time data stream; calculating a dynamic threshold value by adopting a space-time dynamic weight algorithm in combination with historical data pre-aggregated by the Dores star model; multi-dimensional alarm conditions are defined through an SQL-like rule engine, composite alarm triggering of cross-time periods, geographic positions and equipment states is supported, and work orders are automatically distributed to events which are abnormal and last for more than 30 minutes in combination with a network quality perception suppression false alarm and hierarchical upgrading mechanism. The method solves the problems that a traditional scheme is insufficient in real-time performance, single in dimension and rigid in static threshold value.

The invention relates to a metadata-driven data consanguinity analysis and influence evaluation method, system and device and a medium. The method comprises the following steps: performing parameter analysis on an original query language script, and constructing a parameter function model in combination with a parameter dimension table; performing parameter replacement on an original query language script according to the model, and analyzing the replaced script to obtain a table-level blood relationship, a field-level blood relationship and the like; constructing a record-level query template based on the field-level blood relationship and the conversion logic, and constructing a dynamic blood relationship map; and executing a record level query template, carrying out record node mounting on the atlas, obtaining and responding to a data change event, carrying out hierarchical influence propagation calculation, obtaining and responding to a data exception event, carrying out root cause node positioning based on the mounted dynamic blood relationship atlas, and generating a repair suggestion. According to the method, through dynamic parameter analysis and record-level blood relationship analysis, the accuracy and automation level of data blood relationship analysis are improved, and the efficiency of data governance is improved.

The invention discloses an equipment data unified integration and supply method based on dynamic metadata and intelligent routing and an Internet of Things home platform, and the method comprises the steps: obtaining to-be-updated equipment data, classifying the equipment data according to the data structure, data increment and change frequency of the equipment data, and respectively marking the equipment data as main data or transaction data; based on task characteristics of the main data and the transaction data, intelligent scheduling is carried out by utilizing dynamic node pressure sensing, and different types of data are distributed to optimally matched processing nodes; and performing corresponding update processing on the main data and the transaction data, uniformly storing update processing results in a dimension table mode and a fact table mode, and enabling the update processing results to be called by a smart home language model through a uniform data service interface.

The application discloses a meteorological data processing and storage method and system, and relates to the technical field of intelligent meteorological data processing. The method comprises the following steps: collecting multi-source meteorological observation data, radar data and numerical prediction data, performing access determination to obtain access data sets, performing standardization and elevation correction on the access data sets, calculating comprehensive quality scores to obtain quality marked data sets; filtering samples based on the quality marked data sets to obtain aligned data sets, obtaining grid fusion data through the aligned data sets and performing smoothing processing to obtain a fusion data set; obtaining a sealing partition based on the fusion data set, and revising the sealing partition by using a late sample to obtain a versioned sealing partition; obtaining a fact table and a dimension table according to the versioned sealing partition, establishing a subject domain model and a multi-version query interface, and calculating a prediction deviation and a hit rate. Through unit standardization, spatial elevation correction and comprehensive quality scoring on the access data sets, dynamic evaluation and reliable marking of meteorological data quality are realized.

In various examples, table classification based query join reordering for relational database systems and applications are provided. In some embodiments, a relational database system is provided that includes a join optimizer that evaluates a join clause of a query and categorizes relational database tables as either fact tables or dimension tables based on a normalized cardinality statistic. The join optimizer uses the fact and dimension tables to deconstruct the query into a plurality of deconstructed query join trees. Individual deconstructed query join trees may be generated for each respective fact table. The deconstructed query join trees may be joined to generate a reordered join solution representing a sequential join of the plurality of deconstructed query join trees. An updated query may be generated based on the reordered join solution, and a query response generated that answers the query based at least on the updated query.

The invention discloses a data processing method, device and equipment, a computer medium and a product, and is applied to the field of data processing.The method comprises the steps that in response to detected business operation of data change of an upstream business system, business data are collected from a database log; dividing the business data into real-time business data and dimension table business data based on the updating frequency of the business data; sending the real-time business data and the dimension table business data to a streaming computing platform through a database acquisition component; storing the dimension table service data in a preset initial increment dimension table cache through the streaming computing platform to obtain an updated increment dimension table cache; on the basis of the service identification information, associating the real-time service data with dimension table service data in the update increment dimension table cache to obtain service result data; and the streaming computing platform sends the service result data to a data subscription demander through the message middleware. According to the method, data processing efficiency is improved, and computing resources are saved.

This invention discloses a method and system for implementing role-playing dimensions, relating to the field of data processing technology. It includes: creating a meta-model of dimension models related to role-playing dimensions, the meta-model comprising five types of data tables; selecting role-playing dimension names from their sources according to requirements; when a role-playing dimension name differs from a physical dimension table name, the dimension corresponding to the role-playing dimension name is considered a role-playing dimension; when displaying the related dimensions of an indicator, using the role-playing dimension name to replace the physical dimension name to demonstrate the actual business meaning of the dimension; when summarizing the indicator using selected dimensions, constructing dynamic SQL using the role-playing dimension name; when analyzing composite indicators, if the facts of the multi-atomic indicators constituting the composite indicator are all associated with the same physical dimension, then the composite indicator is summarized using that physical dimension; when performing parallel analysis of multiple indicators, if the facts of each indicator involved in the analysis are all associated with the same physical dimension, then the multiple indicators are summarized using the physical dimension; and displaying the list of related dimensions using the physical dimension name to avoid confusion caused by different role names.

The invention discloses a data warehouse model design method, and belongs to the technical field of data warehouse modeling. According to the method, a dimension modeling method is adopted to construct and analyze a mart model, data is organized into a non-standardized relation model, and the relation between tables is defined through keywords and foreign keys. And designing a dimension table system comprising a product dimension, a calendar day dimension, an institution dimension, an insurance policy attribute dimension and a customer dimension, and a fact table system comprising an insurance claim settlement analysis fact table, a payment analysis fact table, a claim settlement efficiency analysis fact table and a corresponding intermediate fact table. The analysis subject domain data is formatted into a star model or snowflake model, where the analysis fact table is a lightly summarized transactional fact table, and the intermediate fact table is a versioned historical table. According to the method, repeated data construction is avoided through a unified modeling method, and the cost is reduced; an efficient query analysis function is provided by optimizing a model structure; and the data traceability is ensured through versioning management.

This application provides a method, device, and storage medium for dimension table joins in Flink, relating to the field of big data technology. The method identifies a target subtask among multiple subtasks of a unified dimension table join operator in the current task manager. After controlling the target subtask to send a query request to an external database and obtain the dimension table data sent by the external database, the method controls the target subtask to load the dimension table data into a static dimension table object set in the metadata space of the task manager. This avoids putting significant operational pressure on the external database in a short period. The method controls the target subtask and multiple other subtasks to perform dimension table join operations based on the obtained raw data and the dimension table data in the static dimension table object, allowing all subtasks in the same task manager to share the same dimension table data, thereby saving physical memory resources.

The present application provides a method, a computing device, and an electronic device for computing stream data. The method includes: obtaining the primary key value corresponding to the target stream data and the primary key value corresponding to the dimension table, and obtaining the target stream data primary key value and the dimension table primary key value; determining the first target partition of the dimension table according to the partitioning strategy based on the dimension table primary key value, and obtaining the first mapping relationship between the first target partition and the node, determining the target node corresponding to the first target partition according to the first mapping relationship, and storing the dimension table to the target node; determining the second target partition of the target stream data according to the partitioning strategy based on the target stream data primary key value, and obtaining the second mapping relationship between the second target partition and the node, determining the target node corresponding to the second target partition according to the second mapping relationship, and storing the target stream data to the target node, so as to perform streaming computing on the target stream data and the dimension table on the target node. This solves the problem of decreased stream data computing performance caused by storing a large number of dimension tables on the same node.

The embodiment of the application provides a data query method, system, device, equipment and storage medium, and relates to the technical field of data processing. The specific implementation scheme is: receiving a first statistical query statement, determining a specified statistical range, a specified statistical method and a specified measurement indicated by the first statistical query statement; if there is no aggregate table containing to-be-queried data in each aggregate table stored in advance, determining whether there is a target aggregate table in each aggregate table, the statistical range of which belongs to the specified statistical range, the statistical measurement of which is the specified measurement, and the statistical method of which is the specified statistical method based on the hierarchical structure between each dimension table stored in advance; if there is, obtaining the measurement value of the specified measurement corresponding to each statistical range belonging to the specified statistical range from the target aggregate table based on the hierarchical structure, and obtaining a query result based on the obtained measurement value and the specified statistical method. It can be seen that, by the scheme, the query efficiency can be improved.

The invention discloses a data warehouse construction and application method based on multi-source data fusion, and relates to the technical field of data fusion, and the method comprises the steps: multi-source data integration layer construction: obtaining structured, unstructured and semi-structured data through a heterogeneous data collection module, carrying out the automatic preprocessing and standardization processing of the data, and obtaining a multi-source data integration layer; the standardization processing comprises a unified naming rule, a coding rule and a data type; designing a core model of the data warehouse: constructing a star model based on a business theme, the star model comprising at least one fact table and a plurality of associated dimension tables, the fact table storing business index data, and the dimension tables storing descriptive attribute data; implementing an ETL automatic process: configuring a data extraction and conversion rule and a scheduling strategy through a visual ETL tool, automatically loading the data processed by the integration layer to a star model, and monitoring a task state in real time; and intelligent query and application recommendation.

The invention discloses a method and a device for generating a commodity pool based on a new machine recommendation algorithm of big data. The method comprises the following steps: constructing a basic view through a Spark framework; performing popularity quantitative evaluation through a commodity popularity scoring model, and respectively screening out hot-sell commodities and new commodities; performing data matching through an SQL linked table query operation, and generating an initial candidate commodity set and a quantity target value of each type of commodities; calculating to obtain a comprehensive score through a multi-objective optimization model; partition-level commodity filtering and optimization are carried out through a MapReduce programming model, and an optimized commodity candidate pool is obtained; performing commodity compliance verification according to items of the white list, and based on a forbidden list in the dimension list black list, adding just-needed categories according to scene requirements through reverse connection operation to obtain a final commodity candidate pool; and merging the commodity data in the final commodity candidate pool into a hive table for persistent storage. The problems that in the prior art, consumption behavior recognition is lacked, and the requirement of a special scene cannot be met are solved.

The invention provides a three-phase asynchronous motor database construction method and system, and the method comprises the steps: obtaining the source data of a three-phase asynchronous motor, and the source data comprises the common parameters, geometric dimensions and performance parameters of the motor; analyzing a data dependency relationship among the common parameters, the geometric dimensions and the performance parameters to determine a determination condition of each data set; creating a plurality of correlative data tables based on the data dependency relationship, wherein the data tables comprise a geometric dimension table group for storing geometric dimensions of the motor and a performance parameter table for storing performance parameters of the motor; and constructing an entity-relationship model of the database based on the association relationship among the plurality of data tables, and generating an SQL statement for data query. According to the technical scheme provided by the invention, structured and standardized storage of the motor data is realized, data redundancy is remarkably reduced, data reusability and query efficiency are improved, and industrial digital application is effectively supported.

The embodiment of the application provides a kind of data table classification method, device, equipment and storage medium, it is related to big data processing technical field, the method comprises: from multiple preset keywords determine at least one target keyword matched with the dimension table information of data table, for any target keyword, based on classification label corresponding relationship, determine target classification label corresponding to target keyword, and the multidimensional chi-square value associated with target classification label and target keyword.Finally, based on the target classification label corresponding to each target keyword and the multidimensional chi-square value associated with target classification label and each target keyword, determine the classification result of data table, instead of relying on artificial experience, improve the accuracy of data table classification result, in turn guarantee the accuracy of data table division to corresponding data warehouse hierarchy.

The invention provides a wide table construction method, a wide table construction system and related equipment, and relates to the wide table construction method and the wide table construction system. A user with a wide table construction demand only needs to input a wide table construction instruction carrying wide table construction related information (such as an identifier of a fact table, an identifier of a dimension table, an identifier of a column of a to-be-added wide table in the fact table, an identifier of a column of the to-be-added wide table in the dimension table and the like) to a wide table construction system; in this way, the wide table construction system can automatically achieve wide table construction according to the wide table construction instruction input by the user and provide the wide table for the user, the whole wide table construction process is simple and rapid, and the construction efficiency is high.

In various examples, table classification based query join reordering for relational database systems and applications are provided. In some embodiments, a relational database system is provided that includes a join optimizer that evaluates a join clause of a query and categorizes relational database tables as either fact tables or dimension tables based on a normalized cardinality statistic. The join optimizer uses the fact and dimension tables to deconstruct the query into a plurality of deconstructed query join trees. Individual deconstructed query join trees may be generated for each respective fact table. The deconstructed query join trees may be joined to generate a reordered join solution representing a sequential join of the plurality of deconstructed query join trees. An updated query may be generated based on the reordered join solution, and a query response generated that answers the query based at least on the updated query.

The application provides a metadata analysis processing method and device, and relates to the technical field of big data analysis processing.The method comprises the following steps: collecting metadata in a data source; acquiring a business process, declaring the granularity of a metadata model according to the business process, and constructing a dimension table and a fact table according to the granularity; taking the business process as a modeling driver, and establishing a metadata model according to the metadata, the dimension table and the fact table; when a data analysis request is acquired, performing data analysis by using the metadata model to obtain an analysis result.The metadata model is established by collecting metadata, the metadata of enterprises scattered in multiple application systems is centrally managed, a complete and comprehensive data structure relationship is presented for the enterprises, the enterprises are assisted in centrally and efficiently managing data assets, the metadata analysis processing of different business requirements can be met, data value can be mined from the analysis result to guide business or product innovation, and the production of enterprises is empowered.

The present application provides a data analysis method, data analysis device, and data analysis system for multidimensional data. The method includes: obtaining a target data table, analyzing the target data table to obtain a fact table and multiple dimension tables; analyzing the fact table and each dimension table to obtain multiple dimension fields and multiple metric fields; analyzing each dimension field to determine a first association type between each dimension field and combining the dimension fields according to the first association type to obtain multiple first dimension groups; adding the metric fields corresponding to the dimension fields in each first dimension group to the corresponding first dimension group to obtain multiple second dimension groups, and constructing corresponding data models based on each second dimension group to obtain multiple cubes; and performing data analysis on each cube to obtain multiple data analysis results. This method solves the problem in the prior art of excessive computational complexity and excessive resource consumption caused by combining all dimensions when performing data analysis on a data model.

The application discloses a protective shoe AI generation method based on physical process constraints and safety standard verification, relates to the technical field of product design automation, and comprises the following steps: assembling the topological relationship of the protective component and the material process constraint based on the scale conversion benchmark and the determinable rule table, binding the rules according to the functional partition, forming a parameterized constraint set, generating a shoe scheme according to the parameterized constraint set and the appearance intention, performing semantic segmentation, key point positioning and edge extraction measurement and verifying the rules, when the verification result indicates that the shoe scheme does not conform to the determinable rule table and the parameterized constraint set, sequentially performing local shape correction, local redrawing generation and discarding regeneration, and obtaining a final scheme, rechecking the final scheme, outputting a multi-view design drawing, a partition annotation drawing, a vectorized contour line, a key dimension table, a material list and a process specification, and performing encryption and decryption, clearing the plaintext and recording the access log.

This application discloses a method for constructing a data model, a method for data querying, an apparatus, and a storage medium, relating to the field of data processing technology. The method includes: obtaining field information from a dimensional model and at least two dimensions of a dimension table, where the field information indicates a mapping relationship between at least two dimensions; constructing a full dimensional relationship graph based on the field information, where the field information also indicates the association relationship between each of the at least two dimensions and a fact table; and attaching the fact table to at least one dimension in the full dimensional relationship graph based on the field information, generating a directed acyclic connected graph centered on the fact table as a data model, used to determine the path from the query metric to the fact table based on the query metric and its dimensions. This method can construct a data model based on existing data warehouse construction achievements, thereby determining the path of the query metric in the data warehouse through the data model.

The invention discloses an elastic dimension modeling method oriented to intensive storage and rapid retrieval, and relates to the technical field of data warehouses and big data, a hierarchical dimension architecture is adopted, dimensions are divided into core dimensions and extension dimensions, access attributes can be dynamically loaded according to service requirements, the extension dimensions are dynamically loaded through metadata driving, and the service quality is improved. The redundancy of core dimension tables is reduced, the storage efficiency is improved, a dimension merging strategy is adopted, the core dimension tables are merged into composite dimensions, the number of the dimension tables is reduced, the query performance is improved, meanwhile, stored data are efficiently compressed, the occupied storage space is reduced, time partition and Hash bucket dividing strategies are adopted for optimizing fact tables, time range query is accelerated, and the query efficiency is improved. The high-frequency access dimension table is loaded to a memory database such as Redis for caching, query response is accelerated, parallel processing is conducted on large-scale data through a distributed query engine, and query efficiency is improved.

The application relates to the fields of smart medical treatment, artificial intelligence and financial technology, and discloses a business data query method and device, a storage medium and electronic equipment. The method comprises the following steps: performing data extraction based on pre-acquired source data tables, obtaining each business table corresponding to each business node of a business process and each dimension table corresponding to the business process; constructing a mapping relationship between each field of each business table and each theme keyword of each dimension table; performing splitting processing on each business table to generate a plurality of sub-business tables corresponding to each business table, and storing each sub-business table; and querying each mapping relationship and each sub-business table based on a user's query request to obtain a query result. The method can improve query efficiency and save query cost.

The invention discloses a method, device and equipment for generating an SQL based on a natural language and a storage medium, and relates to the technical field of artificial intelligence and databases. The method comprises the following steps: firstly, establishing a knowledge graph to clarify an incidence relation between an index and a fact table and an incidence relation between the fact table and a dimension table, then identifying and locking a target fact table through a target index, carrying out correlation retrieval to form a candidate dimension table set, and analyzing and screening a step-by-step flow of the target dimension table through semantics and limitation conditions. The context input into the large language model is always a minimum complete set related to query, the problem of context window limitation caused by a full-scale table structure in a large-scale database is effectively avoided, meanwhile, interference of irrelevant table structure information is reduced fundamentally, retrieval errors caused by same-name fields are avoided, and the retrieval efficiency is improved. The dimensionality semantic ambiguity in the natural language instruction is accurately solved, so that the generated SQL statement can be accurately matched with the query intention of the user, and good performability is achieved.

The invention provides a laboratory data analysis method, and belongs to the technical field of data analysis, and the method comprises the steps: obtaining multi-modal original data of a laboratory based on a locked edge node, carrying out the preprocessing of the multi-modal original data, mapping the multiple preprocessing results into a preset multi-dimensional table, and carrying out the mapping of the multi-dimensional table according to a theme correlation variable; performing false analysis on the action data and the result data after each preprocessing to generate a false label set, and labeling; constructing a project reference matrix of the same experiment project and a step reference matrix of each experiment step, and combining the global feature vector and the local feature vector of each experiment step to determine abnormal type distribution and abnormal result distribution of abnormal operation of each experiment step, and converting the abnormal type distribution and the abnormal result distribution into abnormal vectors; and inputting into a pre-trained analysis model to obtain an equipment regulation and control suggestion, and performing instruction application. And the reliability of experimental data, the comprehensiveness of anomaly recognition and the effectiveness of equipment regulation and control are remarkably improved, so that the experimental efficiency and the result accuracy are improved.

The invention discloses a dimension construction application method and device compatible with coarse and fine granularity, and relates to the technical field of big data processing. Comprising the following steps of: 1, reconstructing a dimension table structure, adding proxy keys of each granularity, and aligning each attribute of a dimension with each proxy key granularity; 2, expanding the dimension table element model, adding granularity level description information, and marking granularity levels of all proxy keys and attributes; 3, when the fact table and the dimension table are established to be associated, according to the actual granularity of the business facts, selecting the proxy key of the corresponding granularity in the dimensions to be associated; step 4, performing de-duplication on a dimension table coarse-grained proxy key, associating a fact table with a de-duplication data set, realizing accurate cross-granularity statistical analysis, and clearly prompting the change of the meaning of an index service; according to the method, the limitation that facts and dimensions must be consistent in granularity is broken through, granularity difference between a fact table and a dimension table in dimension modeling is allowed, and the requirement for grouping statistics of the facts by using dimension attributes with finer granularity is met.