26 results about "Natural key" patented technology

The invention discloses an intelligent database switching method and device, computer equipment and a storage medium, belongs to the technical field of big data, and is applied to a financial database downtime processing scene. According to the method, the node state of the database is monitored in real time, multi-dimensional load evaluation is combined, the optimal main and standby database combination is automatically selected, second-level fault sensing and rapid switching are achieved, and a minute-level time window needed by traditional database recovery is remarkably shortened. The database identifier is embedded in the service main key, so that the request can accurately position the corresponding database node, complex cross-database query and routing overhead are avoided, and the system response efficiency is improved. In addition, through data seamless transmission and automatic abnormal switching between the main database and the standby database, the risk of service interruption caused by database faults is reduced. According to the scheme, the high availability and the fault recovery speed of the database are improved, repeated construction of a multi-product-line database cluster is reduced through resource integration, and a large amount of hardware and operation and maintenance cost are saved.

This application relates to automatic processes that service data requests for information related to an item during varying temporal periods. For example, a computing device receives first attributes for an item, and generates a first surrogate key and a first super surrogate key based on the first attributes. Further, the computing device generates a first entry within a first table that includes the first surrogate key and the first super surrogate key. The computing device joins the first table to a second table based on natural keys. The computing device also receives second attributes for the item, and generates a second surrogate key for the item based on the second attributes. Further, the computing device generates a second entry within the first table that includes the second surrogate key and the first super surrogate key. The computing device also adjusts the first super surrogate key within the first entry.

This application discloses a data processing method, apparatus, device, and computer-readable storage medium. The method includes the steps of: acquiring collected data and generating a business primary key for the collected data; the business primary key having resource uniqueness; comparing the differences between the collected data and existing data based on the business primary key to obtain a comparison result; and processing the existing data accordingly based on the comparison result. This application enables accurate comparison of collected data and existing data through the business primary key, thereby regaining the ability to manage collected data and preventing collected data from slipping out of control.

The application discloses a multi-type property maintenance work order automatic generation and management method and system, and the method comprises the following steps: triggering a timing task according to the execution time of a maintenance plan, and judging whether the maintenance plan is overdue; when the maintenance plan is not overdue, creating a work order record and writing the work order record into a business data table, and starting a work flow instance with a business primary key in the same database transaction; the work flow instance triggers an execution listener in a node flow process, updates a work order processing state, and executes corresponding business logic according to a node identifier; a timing job record is written in a user task node, a work order timeout mark is updated and a timeout alarm is sent when a timeout deadline arrives, so that the work order automatic generation, process linkage and timeout control are realized.

This application provides a method, apparatus, device, and storage medium for verifying data table consistency. The method includes: acquiring first business primary key information and a first identifier corresponding to multiple first data tables in design states, and second business primary key information and a second identifier corresponding to multiple second data tables in production states; comparing whether the first business primary key information and the second business primary key information are consistent; for first data tables and second data tables with consistent business primary key information, comparing whether the first identifier and the second identifier are consistent; for first data tables and second data tables with consistent identifiers, comparing whether the fields of the first data table and the fields of the second data table are consistent; and determining the verification result of data table consistency based on the primary key information comparison result, the identifier comparison result, and the field comparison result. The verification method according to this application embodiment can automatically verify whether the structures of physical model tables and database tables are consistent and obtain relatively accurate verification results.

The embodiment of the invention provides a data quality inspection method based on dynamic parameter transmission and related equipment, and belongs to the technical field of data management. The method comprises the steps that a quality inspection scheme is newly established, the quality inspection scheme comprises a quality inspection scheme type and a data source, and the quality inspection scheme type comprises a table conventional quality inspection type and a consistency quality inspection type; based on the quality inspection scheme type and the data source, performing quality inspection rule configuration on the quality inspection scheme to obtain a quality inspection scheme after rule configuration, the quality inspection scheme after rule configuration comprising a quality inspection rule; configuring a result list and a service main key of the quality inspection scheme; configuring a scheduling period; and executing the quality inspection scheme based on the scheduling cycle, and generating a quality inspection result based on the quality inspection scheme type, the quality inspection rule, the result detail list and the service main key. According to the embodiment of the invention, the quality inspection scheme can be automatically and periodically executed, the working efficiency and the data quality management efficiency can be improved, the accuracy and effectiveness of the data are improved, and a powerful basis is provided for subsequent data recovery.

The invention provides an asynchronous task processing method based on persistent queue and business rule conflict detection, and the method specifically comprises the steps: receiving an e-commerce data flow, and analyzing the e-commerce data flow into an atomic data line object; extracting a to-be-operated resource dimension in the object, and forming a resource locking identifier set consisting of a service main key and an association key; establishing a conflict dependency relationship between write operations, and constructing a directed acyclic dependency graph according to the conflict dependency relationship; the objects are decomposed into a plurality of batches which are executed in sequence, and it is ensured that resource identification sets of all the objects in the same batch are not intersected; for each object, reading the current data version number of the target write-in position, and generating a write-in instruction containing an expected version; verifying the consistency of the real-time version and the expected version by using a database CAS instruction; if the real-time version and the expected version are consistent, finishing persistence; if not, the concurrency version conflict is judged; at the moment, a conflict avoiding mechanism is triggered, rollback time delay is calculated through an exponential back-off algorithm, and conflict objects are re-injected into the dependent construction or execution process.

The application relates to the technical field of data processing, and particularly provides a data fusion method and system based on a data center, which specifically comprises the following steps: acquiring real-time service data and offline service data generated by a source end by the data center respectively; configuring fusion parameters containing an alignment time window, a time tolerance threshold and a service primary key; performing unified window mapping on the time stamps of the two types of data based on a space-time alignment scheduling rule, screening to-be-fused data in the same alignment time window; asynchronously associating the to-be-fused data with offline historical data according to the service primary key and integrating the to-be-fused data into corresponding data sets; completing conflict determination and data merging through preset determination rules, marking data sources and version traceability labels to obtain a fused data stream; and outputting the fused data stream to a target storage medium and packaging the fused data stream into a standardized data interface. The application can realize effective fusion of real-time data and offline data, and improve data consistency and reliability.

The embodiment of the invention provides a big data integration method and device, equipment and a storage medium, and the method comprises the steps: determining to-be-processed data from to-be-integrated data; wherein the to-be-processed data comprises first data, second data and third data; the first data is determined according to timestamps corresponding to all specified service stages in the to-be-integrated data; the second data is determined according to a service main key corresponding to each specified service behavior in the to-be-integrated data; the third data is determined according to a predetermined common universal dimension; performing data conversion on the determined to-be-processed data according to a preset data conversion process; the data conversion process at least comprises one of code value standardization processing, data desensitization and data length standardization processing; and loading the converted data to a target database.

The invention belongs to the technical field of data statistics, and discloses a data statistical method, device and equipment and a storage medium, and the statistical method comprises the steps: processing obtained original business data, and obtaining list data with a business main key; inputting the list data to a preset analysis database, and mapping the service primary key into a unique constraint field of the list data; and when the analysis database receives the incremental list data, judging whether the first service primary key corresponding to the incremental list data has a matched second service primary key in the unique constraint field, and if so, updating the original field in the incremental list data to the list data corresponding to the second service primary key. And if not, mapping the first service primary key to the unique constraint field. The method can be applied to business management systems of medical health, financial science and technology and the like, and solves the technical problem that in the prior art, when a distributed stream processing engine is used for processing data, keys which are updated infrequently are cleared away, and consequently the data statistical result is abnormal.

The application relates to the technical field of big data processing, and discloses an internet operation and sales data analysis method and system based on big data, which comprises the following steps: acquiring multi-source business data records and extracting business primary keys; based on the business primary keys, the multi-source business data records are collected into corresponding logical containers; the logical containers are subjected to integrity evaluation to determine whether the logical containers meet preset business logic integrity requirements; only when the business logic integrity requirements are met, is the associated analysis of the collected data in the logical containers triggered, and corresponding operation analysis conclusions are generated; when the logical containers receive new business data records, the operation analysis conclusions are corrected according to the new business data records. The application constructs an event-driven analysis unit through the business primary keys and the logical containers, solves the pseudo-association and missing-association problems caused by the inconsistent arrival times of multi-source data, and realizes accurate and dynamically-updatable operation insight.

The present application relates to the technical field of data asset management, in particular to a main data coding method, system, device and medium based on business primary key; the method first establishes a configuration table according to the obtained main data object, main data business primary key and business stage; then establishes a coding rule library according to the main data business primary key and the established coding meta model; finally registers the main data model to the main data system, increases the process activity configuration in the maintenance process configuration link, configures the business primary key attribute value after the verification to the coding rule, and generates the main data coding; the flexible coding of the main data object is realized, the main data coding is ensured to be associated and penetrated in each business environment of discrete manufacturing industry, and the penetration of end-to-end process is realized.

The embodiment of the invention discloses a real-time processing method and device for multi-service system data and a computer program product, and the method comprises the steps: obtaining target index data after data change through monitoring the change condition of index data in a plurality of service systems; wherein the plurality of service systems correspond to different stages in a data processing flow; determining a target primary key corresponding to the target index data according to a service primary key corresponding to the target index data; determining a target data entry corresponding to the target index data from a preset wide table corresponding to the data processing flow by taking the target primary key as an index; and updating the target data entry according to the target index data. According to the embodiment of the invention, the real-time sensing, processing and storage of the data are realized, and the real-time performance and accuracy of the data are improved.

A batch process operates on structured data in multiple tables, including a first and a second source table, and a first and a second processing table. An additional key is added within each natural key to at least the second source and processing table for identifying a sequence of rows for updating.

The invention discloses an incremental data detection method and device, equipment and a storage medium, and relates to the technical field of incremental data detection.The incremental data detection method comprises the steps that a service primary key and a data fingerprint are generated for each data record in a heterogeneous data source; constructing a data snapshot according to the business primary key of each data record and the corresponding data fingerprint; comparing the data snapshot with a stored historical data snapshot to identify a data change record; and outputting an incremental data result set containing the data change record. According to the method, the universality of incremental data detection can be improved.

The invention discloses a reconstruction method and device for an international freight rate database and electronic equipment, the method is applied to the field of data reconstruction, and the method comprises the following steps: carrying out data analysis operation on data in N tasks in parallel through N processes, and generating structured data of each task in the N tasks; grouping the structured data of each task based on the service primary key to obtain M groups corresponding to the N tasks; performing data reconstruction operation on the data in the M groups corresponding to the N tasks in parallel through the M sub-processes to generate M groups of file databases corresponding to the N tasks; and merging the M groups of file databases corresponding to the N tasks according to the partition sequence, and determining a reconstructed international freight rate database according to the merged file database. Through the method and the device, the problem of relatively low database reconstruction efficiency caused by large data volume and need to process data according to a time sequence in the process of reconstructing the international freight rate database by an airline company in related technologies is solved.

The invention discloses a system process series connection method and device and a storage medium, and relates to the technical field of business data processing, and the method comprises the steps: extracting series connection key elements based on a preprocessed business system original log, packaging the series connection key elements, and outputting the packaged series connection key elements as an event data stream; based on the event data stream, performing service primary key deduplication processing, and obtaining a first relation stream of a service primary key according to the event data stream subjected to the deduplication processing; based on the event data stream, extracting a second relation stream associated with a service primary key; combining the first relation flow and the second relation flow into a global incidence relation set; and carrying out data merging on the global association relationship set and a database state table to generate a global process identifier, and carrying out system process tracking. According to the method and the device, the flow series connection of different events in the same system, different events in the same service and cross-system service events can be realized.

The present disclosure belongs to the technical field of data processing, and provides an XML message comparison method and system, a storage medium and an electronic device. The method comprises receiving system messages of different message sending sources, determining a business primary key and corresponding update data; determining initial data corresponding to the business primary key, comparing the update data with the corresponding initial data, and determining whether the update data has a special event; generating an XML message in a preset format according to the update data with the special event and the corresponding business primary key; comparing the XML messages corresponding to the same business primary key and different message sending sources within a preset time range, and determining a comparison result. The XML message is automatically generated when the special event occurs, the XML messages corresponding to different message sending sources are automatically compared, the manual cost is saved, and missing or incorrect judgment is avoided; the difference field position in the XML message can be clearly and accurately located, and the efficiency and accuracy of the message comparison work are improved.

The invention provides a data synchronization method, system and device based on dynamic region configuration and a storage medium, and the method comprises the steps: monitoring a binary log stream generated by a source database in real time through an integrated data replicator, and extracting data change information; pre-configuring and dynamically maintaining metadata, and querying in a configuration center according to the table name to which the data change operation belongs, so as to obtain metadata corresponding to the table name to which the data change operation belongs; and asynchronously calling a business guide service with a business rule engine according to the business main key value, carrying out logic judgment on the business main key value according to the business rule engine, and when synchronization is needed, applying data change information to a corresponding table of a target database in a transaction by a data copy application component deployed in a target data center. According to the invention, data synchronization logic can be decoupled from a service system, a synchronization strategy is managed in a configuration mode, invalid data transmission is reduced by using intelligent decision, and the utilization rate of system resources is improved.

This disclosure provides a method for querying transaction endpoints, applicable to the field of big data technology. The method includes: obtaining the first business primary key information of the transaction to be queried and information about the first business transaction scenario to which it belongs; obtaining the application link configured for the first business transaction scenario to obtain a first application link; determining the transaction identifier information used to identify the transaction to be queried in any first application within the first application link; and determining the endpoint server executing the transaction to be queried by retrieving the transaction identifier information from the logs of S servers in the first application. This disclosure also provides a transaction endpoint query device, equipment, storage medium, and program product.

The invention discloses a data updating method and device, computer equipment, a storage medium and a program product, and belongs to the technical field of databases. The method comprises the steps that new data are written into a first index, a random document identifier is generated for a document where each piece of new data is located, and the new data comprise timestamp fields recording data writing time; querying all the data in the first index, grouping all the data, sorting the data in each group according to timestamp fields of all the data, and screening the data in each group to obtain screened data; and performing an initialization operation on the first index every a first time period. According to the method, data are screened through timestamp sorting and business primary key grouping, and the problems of data updating efficiency and accuracy of Elasticsearch in an online synchronous service scene are solved in combination with periodic index initialization.

The application discloses a data processing method in a stream computing scene and relates to the technical field of data processing, which is executed by a stream computing platform and comprises the following steps: performing deserialization processing on metadata of an original CDC event to obtain structured metadata; determining whether the original CDC event is a deletion semantic event according to a value of an operation type field and a value of a business primary key field of the structured metadata; if the original CDC event is the deletion semantic event, performing semantic conversion processing on the structured metadata according to an event type of the structured metadata to obtain semantic conversion metadata; performing deserialization processing on business data of the original CDC event to obtain structured business data; and generating a RowData event according to the semantic conversion metadata and the structured business data and sending the RowData event to a downstream application for consumption. The application realizes semantic conversion of the original CDC event in the stream computing platform between an upstream database and a downstream application, avoids repeated conversion of the downstream application, and improves the processing efficiency of the event.

The invention provides a service primary key generation method and device, and relates to the technical field of data processing. The method comprises the following steps: receiving a service primary key request sent by an application system; obtaining primary key configuration information from a memory database according to the primary key ID, and generating a new primary key value according to the primary key configuration information and a preset primary key generation mode; and taking the new primary key value as a current primary key, returning the current primary key to the application system, and storing the current primary key to the memory database so as to synchronize the current primary key to the relational database by calling a data synchronization service. The device executes the method. According to the service primary key generation method and device provided by the embodiment of the invention, the service primary key of the self-increasing sequence can be automatically generated in a distributed environment, and the service attributes are expressed through the service primary key, so that the service requirements are fully met.

The invention relates to the technical field of computers, and provides a data table storage method, a data table storage device, a computer storage medium and electronic device.The data table storage method comprises the steps that information of M data tables needing to be compressed is read from a preset compression configuration file; m is an integer greater than 1; obtaining the M data tables according to the information of the M data tables needing to be compressed; compressing and merging the M data tables according to compression and merging demand information contained in the compression configuration file to obtain a compression and merging result; the compression and merging demand information comprises a service main key corresponding to each data table in the M data tables and at least one to-be-compressed field corresponding to each data table; and recompressing the compression and combination result, and storing the obtained recompression result to a specified storage system. According to the method, the data storage performance can be improved.

The invention discloses a database table entity conversion method, computer equipment, a medium and a product, and relates to the field of database design. By implementing the method, the computer equipment traverses the database tables, extracts the structured data of each field in each database table, establishes the field association mapping table based on the field identifier, the field name, the service primary key, the logic foreign key and / or the dictionary code, and effectively identifies and manages the logic relationship between the tables. And the computer equipment uniformly processes the logic foreign key relationship and dictionary coding mapping, so that the data integrity is ensured, and the flexible data translation capability is provided. Moreover, when the computer equipment is converted into the database design entity and the data transmission object, business semantics and UI display requirements are completely reserved, so that the converted data transmission object not only conforms to database design specifications, but also is convenient to apply, develop and use, and the database design and development efficiency is improved.

The invention relates to an enterprise risk data analysis method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring enterprise risk data and a business main key thereof; analyzing a pre-configured analysis rule pool according to the enterprise risk data and the business major key thereof, and generating a plurality of target scene rules sorted according to priorities; according to priority ranking of the multiple target scene rules, whether the enterprise risk data meet main judgment conditions corresponding to the target scene rules or not is judged in sequence till the main judgment conditions met by the enterprise risk data are determined; and determining a risk reason of the enterprise risk data according to the scene label of the target scene rule satisfied by the enterprise risk data, and marking the enterprise risk data. According to the scheme provided by the invention, the analysis rules can be flexibly configured, the risk data of different risk classes and different business scenes can be analyzed, rapid configuration and intensive management of the analysis rules can be realized, and the risk data processing efficiency and accuracy can be improved.