31 results about "Data definition language" patented technology

The invention discloses a data warehouse table structure generation method and device, a medium and a product, and belongs to the technical field of data warehouses. According to the method, the Excel template configured with multiple pieces of source database table structure information is obtained, table field definition is automatically analyzed, the source data type is intelligently mapped and converted into the target data type according to the source database type, meanwhile, partition key fields are automatically recognized based on the preset strategy, and finally a complete and accurate data definition language (DDL) statement is generated. According to the method, remarkable technical breakthrough and value are achieved in practical application, the table building efficiency can be improved by 90% or above, batch processing of hundreds of tables is supported, and manual errors are reduced by 95% or above; mainstream database type conversion is comprehensively supported, and various data models such as Dores are adapted; the method has the capabilities of automatic type detection and conversion, intelligent key field selection and partitioning and bucket dividing strategy optimization.

The embodiment of the present specification provides a DDL statement execution method and device, and relates to the technical field of databases.The method comprises the following steps: in response to an execution request for a data definition language (DDL) statement, starting a transaction, the transaction being used for executing the DDL statement; recording target lock information corresponding to the DDL statement in the transaction, the target lock information being used for indicating parameters of locks added on a database object in the execution process of the DDL statement; in response to successful submission of the transaction, deleting the target lock information, so as to complete the execution of the DDL statement.The scheme provided by the present specification connects the lock operation in the execution process of the DDL statement into the transaction, provides a perfect lock management mechanism, makes the execution condition of the DDL statement consistent with the control of the lock, and thus can coordinate the concurrent processing of multiple DDL statements.

This invention discloses a real-time data synchronization method based on DAG orchestration and asynchronous zero-copy architecture, comprising: constructing a directed acyclic graph (DAG) based on timestamped data and its data lineage to obtain parallelizable task nodes; sending data using a zero-copy transmission method based on io_uring based on the parallelizable task nodes to obtain zero-copy transmitted data; grouping the received data according to the source IP address and storing it in the corresponding receiving queue to obtain grouped queue data; and sorting and merging the grouped queue data according to the global timestamp using a time window multi-path merge algorithm, and using an event barrier mechanism to handle the order of data definition language instructions and data manipulation language instructions to obtain a globally ordered event stream. This invention achieves dependency-aware parallel scheduling, zero-copy asynchronous transmission, and globally ordered merging of cross-source events.

The application discloses an implementation method and device for online data definition language operation of a graph database. The features include: receiving an online data definition language operation on the graph database, executing the online data definition language operation, determining a to-be-updated graph data version corresponding to the online data definition language operation, identifying a database operation type corresponding to the online data definition language operation, generating a schema update task in the case that the database operation type or a version control system detects that the graph database meets a preset graph database update condition, and updating the to-be-updated graph data version to the graph database according to the schema update task. The application supports continuous business operation in the schema change process, minimizes system downtime, reduces redundant storage, reduces storage cost, provides strong support for fault recovery and problem diagnosis through a version control mechanism, supports more efficient distributed synchronization, and reduces network overhead and computing resource consumption.

The invention relates to a data processing method and device, equipment and a storage medium, and the method comprises the steps: obtaining a to-be-recognized SQL file and a corresponding configuration file, the configuration file comprises first corresponding information, and the first corresponding information comprises a corresponding relation between a table field and a configuration constraint; according to the SQL file, second corresponding information and related table names are determined, and the second corresponding information comprises table fields and corresponding information of initial constraints; extracting third corresponding information from a data definition language file corresponding to the related table name, wherein the third corresponding information comprises table fields and corresponding information of supplementary constraints; determining a legal value corresponding to each table field according to the first corresponding information, the second corresponding information and the third corresponding information; and executing database operation on the SQL database according to the legal value corresponding to each table field. According to the method, abnormal value insertion can be effectively avoided, so that deviation of data statistics is prevented.

The invention discloses a data synchronization method, device and equipment and a computer readable storage medium, which are applied to the technical field of data processing, and the method comprises the following steps: converting a data definition language event and a data manipulation language event into a data definition structured statement and a data manipulation structured statement; determining whether the data manipulation structured statement depends on the data definition structured statement or not by utilizing a dependency analysis model; and when the data manipulation structured statement does not depend on the data definition structured statement, determining to synchronously execute the data definition language event and the data manipulation language event. Compared with the condition that one data definition language event blocks all data manipulation language events during current data synchronization, the method and the device have the advantages that the dependency relationship between the data definition language event and the data manipulation language event can be determined through the dependency analysis model; therefore, the data definition language event and the data manipulation language event can be synchronized at the same time in the target library, and the data synchronization efficiency can be improved.

The invention relates to the technical field of information processing, and discloses a processing method and device for analyzing JSON data into offline table data, a storage medium and a product, and the method comprises the steps that task configuration is read; obtaining original data according to the task configuration, and generating a first elastic distributed data set based on the original data; performing recursive analysis conversion on the first elastic distributed data set to generate a second elastic distributed data set containing a key value pair set; obtaining an existing field set in the target table; executing distributed calculation on the second elastic distributed data set to obtain a newly added key list; generating a data definition language statement according to the key list and the mode information; executing the data definition language statement to extend the table structure of the target table; and the second elastic distributed data set is converted into the data frame according to the table structure of the expanded target table, and the data frame is written into the target table, so that the JSON data can be automatically analyzed to the offline table, and the processing timeliness of the data is improved.

Embodiments of the present application disclose an event processing method, device and equipment, and a storage medium. The method comprises: obtaining a data update event of a target upstream sub-table in an upstream database when data is updated; in response to the target upstream sub-table not being in a preset state and no data update event being executed for a downstream database, determining whether the data update event comprises a data definition language (DDL) event or a data manipulation language (DML) event; in response to the data update event comprising a DML event, executing the data update event for the downstream database; and in response to the data update event comprising a DDL event, determining a compatibility state of a target table structure before data is updated in the target upstream sub-table and table structures of other upstream sub-tables in the upstream database, and processing the data update event based on the compatibility state. The embodiments of the present application can flexibly process data update events corresponding to upstream sub-tables in an upstream database, and have high applicability.

The present application relates to the technical field of database, and discloses a database rollback method and device, electronic equipment and storage medium, the method comprises the following steps: obtaining a first table structure corresponding to a first database and a second table structure corresponding to a second database; when the first table structure is different from the second table structure, determining the table structure difference information between the first table structure and the second table structure; generating at least one change statement in the form of data definition language based on the difference attribute corresponding to the table structure difference information; if a first change statement fails during the execution of the table structure change process, obtaining a second change statement that has been successfully executed; based on the rollback strategy corresponding to the second change statement, restoring the second database to the original state before the change. By implementing the method, the table structure and business data of the second database can be restored to the original state before the change, thereby not affecting the normal operation of related businesses.

The embodiment of the application provides a kind of database cluster in standby machine temporary table processing method, equipment, storage medium and program product, it is related to distributed field.The method comprises: receiving data definition language (DDL) operation request;In response to determining that the DDL operation request is the request of creating standby machine local temporary table, the pre-write log (WAL) level of DDL operation is set to the log level specially used for the creation of standby machine local temporary table;Local temporary table is allocated identifier, and the identifier includes object identifier and the cluster node identifier of backup node;The metadata of local temporary table is stored in the memory storage area locally of backup node, and the memory storage area is independent of the physical system table synchronized with master node.The method of the application increases the ability of standby machine to independently process local temporary data operation, and improves the utilization efficiency of overall resources of database cluster.

The invention relates to an SQL (Structured Query Language) statement generation method and device based on multi-stage and retrieval enhancement. The method comprises the following steps: respectively inputting a natural language question into a data definition language selector and an object selector; performing semantic analysis on the natural language question to obtain at least one database table related to the natural language question; performing field value matching processing on the natural language question to obtain a standard field value matched with the natural language question; inputting the at least one database table and the standard field value into a demand extraction module to obtain a first output result of the first large model; the natural language question matched with the intention is input into a pre-trained second large model, a second output result of the second large model is obtained, and the second output result is the target SQL statement. According to the scheme provided by the invention, the reliability and accuracy of the SQL statement can be improved through the target SQL statement when the natural language query is converted into the SQL query.

The invention discloses an NL2SQL method, device and equipment based on retrieval enhancement, a medium and a program product. The method comprises the following steps: constructing a three-layer tree index and a database mode abstract of a target database based on a DDL (Data Definition Language) of the target database; wherein the three-layer tree index comprises a surface layer index, a column layer index and a value layer index; constructing an external knowledge base, wherein the external knowledge base comprises knowledge related to the target business corresponding to the target database; under the condition that a natural language question used for querying the target database is received, candidate objects related to the natural language question in the target database are determined based on the three-layer tree-shaped index; acquiring knowledge related to the candidate object from an external knowledge base as enhanced information; and generating a structured query language (SQL) corresponding to the natural language problem based on the natural language problem, the candidate object, the database mode abstract and the enhanced information through a large language model. According to the embodiment of the invention, the accuracy and efficiency of NL2SQL conversion can be improved.

The invention relates to a multi-source data fusion concrete industry single project cost income accounting method, which belongs to the technical field of data processing, and comprises the following steps: S1, collecting and supplementing multi-source heterogeneous data; s2, data structuring is carried out, and a unified language system is established; s3, cross-domain data intelligent association is carried out; s4, ETL processing and automatic binding of multi-source data are carried out; and S5, carrying out visual display on the processed and associated data through a BI report. Through data definition language and natural language processing, a unified data model covering core business entities and key links and a cross-system data dictionary are constructed, the problem of multi-system data semantic conflict is solved, the logic consistency of cost and income accounting is ensured, and intelligent matching of raw material purchase batches and consumption records is achieved.

The invention relates to the technical field of object relation mapping, in particular to a dynamic object relation mapping method. The method comprises the following steps: S1, defining entities and relationships according to a domain model, and converting the entities and relationships into metadata; s2, the dynamic ORM framework analyzes the metadata to generate DDL statements and executes the DDL statements; s3, the transmission value is converted into memory and database type values, and the dynamic ORM maps the operation command of the domain object into an SQL statement and executes the SQL statement; s4, constructing a query expression tree, mapping the query expression tree into an SQL expression model by the dynamic ORM, and generating an SQL statement; and S5, executing the SQL statement to obtain a result set, and converting the result set back to the domain entity. According to the dynamic ORM technology, through metadata description, dynamic analysis and cache storage, real-time updating of the domain model, automatic generation and execution of DDL statements and synchronization and automatic management of a database structure are achieved, low-code direct definition and management of the domain model are supported, complex codes do not need to be written, and the demand response speed and development efficiency are improved.

The embodiment of the invention discloses a data processing method, electronic equipment and a storage medium, and the method comprises the steps: obtaining an offline DDL (Data Definition Language) operation request which indicates a data table name and a to-be-modified data table structure; determining a source data table according to the data table name, and creating a temporary data table according to the to-be-modified data table structure; migrating data in the source data table to the temporary data table according to the to-be-modified data table structure; in response to a service request received in the migration process, processing the service request according to the source data table; and after migration is completed, switching the temporary data table into a new source data table. According to the embodiment, the DML operation is not blocked in the process of responding to the offline DDL operation request, and the production service is not influenced, so that the service production efficiency is improved.

The invention discloses a cross-database-platform data definition language processing method, medium and equipment, and the method comprises the steps: receiving a database structure definition file written in a unified data definition language which comprises an instruction set independent of a specific database platform; reading the file content line by line, identifying and verifying the legality of the instruction text through a regular expression, and converting the instruction text into a predefined intermediate representation object; according to the type of the target database, a corresponding DDL template is called from a preset template library, the template is filled with structural information contained in the intermediate representation object, and a DDL script compatible with the target database is generated through rendering of a template engine; and finally, running the DDL script, and realizing structure creation or change in the target database. According to the method, unified intermediate representation and template rendering are defined, so that one-time writing of database structure definition is realized, multi-database deployment is automatically adapted, and the development efficiency and the deployment reliability are remarkably improved.

This application provides a data processing method, apparatus, electronic device, and medium, relating to the field of data engineering technology. The method includes: determining a target field based on a first operation frequency and a second operation frequency of each field in a Data Definition Language (DDL) operation flow; calculating the path weights of a set of related paths between the target field and the first field to obtain a path weight set; calculating a comprehensive score for the first field based on the semantic similarity between the target field and the first field and the path weight set; and determining a replacement field based on the comprehensive score. The replacement field is used to replace the target field, and the replacement field includes the first field, a second field, and a third field, wherein the second field is a mixed value generated based on the first field, and the third field is a default value. This improves the flexibility and reliability of data processing and adapts to the ETL data flow requirements in scenarios with high-frequency field changes.

The application discloses a data processing method and device based on a PostgreSQL database, equipment and a medium, and relates to the technical field of PG databases. The method comprises the following steps: creating a preset data processing object in a to-be-processed PostgreSQL database to obtain a target database; the preset data processing object is used for collecting, analyzing and recording data definition languages in the database and is used for specifying a specified library that needs to be subjected to data processing; when it is monitored that a data manipulation language exists and is used for operating the target database, incremental change data generated by the operation is published, and a backup tool deployed on a target equipment is used for backing up the incremental change data according to a preset backup strategy to obtain a backup database; and the target database is recovered by a recovery tool deployed on the target equipment based on the backup database to obtain a recovery database. Through the technical scheme, remote data backup of the specified database can be safely and efficiently performed.

The invention relates to the technical field of text processing, and discloses a multi-agent-based unstructured information structured extraction method, which comprises the following steps of: outputting a candidate data definition language and a multi-table distribution sample through a generator, and carrying out statistical projection on the candidate data definition language and the multi-table distribution sample to obtain discrete statistics; constructing a query plan cost phase change surface based on the statistics, and calculating the steady-state probability of each plan in combination with a Markov steady-state model of a plan cache mechanism; calculating a high-quantile tail delay by using a queuing theory model, and solving a second derivative of the tail delay about distributed perturbation in the minimum carrying direction of the Warisstein space to obtain an optimal transmission bifurcation cache congestion curvature; and finally, a generator is updated by taking the minimum curvature as a constraint, and a table structure is output. According to the method, the performance avalanche risk caused by data perturbation can be effectively reduced, and the robustness of the database is remarkably improved.

This application relates to the field of information processing technology, and discloses a method, device, storage medium, and product for parsing JSON data into offline table data. The method includes: reading task configuration; obtaining raw data according to the task configuration, and generating a first elastic distributed dataset based on the raw data; performing recursive parsing transformation on the first elastic distributed dataset to generate a second elastic distributed dataset containing a set of key-value pairs; obtaining the existing set of fields in the target table; performing distributed computation on the second elastic distributed dataset to obtain a list of newly added keys; generating data definition language statements based on the key list and schema information; executing the data definition language statements to extend the table structure of the target table; converting the second elastic distributed dataset into a data frame according to the extended table structure of the target table, and writing the data frame into the target table, thereby automatically parsing JSON data into an offline table and improving the timeliness of data processing.

The invention relates to the technical field of databases, and discloses a database rollback method and device, electronic equipment and a storage medium, and the method comprises the following steps: obtaining a first table structure corresponding to a first database and a second table structure corresponding to a second database; when the first table structure is different from the second table structure, table structure difference information between the first table structure and the second table structure is determined; generating at least one change statement represented in a data definition language form based on a difference attribute corresponding to the table structure difference information; if the first change statement which fails to be executed occurs in the process of executing the table structure change, obtaining a second change statement which is successfully executed at present; and based on a rollback strategy corresponding to the second change statement, recovering the second database to an original state before change. By implementing the method, it can be ensured that the table structure and the service data of the second database can be recovered to the original state before change, and then normal operation of related services cannot be affected.

Embodiments of the present application disclose a distributed transaction processing method, system and device, which can avoid outdated metadata and transaction processing errors caused by outdated metadata, improve transaction processing efficiency, reduce computing, transmission and storage overhead, and ensure complete fault tolerance and disaster tolerance of the system. The distributed transaction processing method comprises the following steps: firstly, starting a distributed transaction GT and a transaction branch CT of the distributed transaction GT on a computing node by using the computing node according to a command statement, the command statement comprising a data definition language (DDL) statement or a data manipulation language (DML) statement sent by a client, and the local transaction branch CT being completed on the computing node; secondly, obtaining a parsing result by parsing the command statement by using the computing node, the parsing result comprising an internal representation and an execution state of the command statement; and finally, creating or modifying metadata in a local metadata table by using the computing node according to the parsing result, the created or modified metadata being related to the command statement, and the metadata table being stored on the computing node.

The present disclosure relates to a data synchronization method and apparatus based on a database, a medium, a device and a product. The method includes: executing, in the database, a target statement carrying a current data synchronization position, to write the current data synchronization position into a log file of the database through the target statement, the target statement is a statement of a data definition language type; searching in the log file of the database to obtain a target data synchronization position in response to data breakpoint resumption being performed, the target data synchronization position is the current data synchronization position recorded when data synchronization is interrupted last time; and performing data synchronization based on the target data. synchronization position.

The application discloses a data dictionary cache management method, device and equipment of a database, a medium and a program product. The method comprises the following steps: in the process of executing a data definition language, updating a global version number and clearing a data dictionary cache involved in the execution process; in the process of executing a data operation language, if the data dictionary cache is not hit, determining whether to add the constructed data dictionary to the data dictionary cache according to the consistency of the global version numbers before and after the data dictionary is constructed. The above technical feature avoids adding an incomplete data dictionary to the cache, thereby solving data errors caused by dictionary residues and improving the execution performance of DDL and DML in the database.

The invention discloses a data standard preposed coding automatic auditing method and system, and relates to the technical field of data management. In order to solve the problems of low efficiency and easy omission caused by the fact that a traditional data standard depends on manual checking, the adopted scheme comprises the steps that the data standard described by a natural language is converted into a machine-readable coding rule, and the coding rule is stored in a rule base; monitoring a data definition language script submission event in the continuous integration / continuous deployment assembly line, and automatically triggering an auditing process; in the auditing process execution stage, the corresponding coding rule is extracted from the rule base according to the current script, the corresponding coding rule is analyzed and converted into executable check logic, and automatic compliance auditing is executed on the current script based on the check logic; according to the audit result, returning a corresponding signal to the continuous integration / continuous deployment assembly line so as to trigger execution pass or process blocking operation; and a violation report and a matched correction suggestion script are automatically generated during the flow blocking operation. The data quality can be improved from the source.

A method is provided for validating transactions in a system that includes indeterministic ordering of change log events by using deterministic indicia. At least some of the change log events include indeterministic values. The system includes a plurality of nodes. Each node in the system includes an instance of a database used by an application, and a change log associated with the instance of the database used by the application. Each node redundantly executes the same transactions to be validated. Each transaction includes one or more data manipulation language (DML) or data definition language (DDL) operations. Results of the DML or DDL, operations are stored in the change log as change log events, wherein the order of the change log events stored in the change log on one node is in a different order than the order of change log events stored in the change log of at least one of the other nodes. For each transaction executing at each node, indeterministic values of the change log events are pre-converted to deterministic values of the change log events. Deterministic subunits of the change log events are defined. Indicia are then computed for each subunit of the change log events. The subunits are ordered by the computed indicia. Deterministic indicia are then computed for the transaction from the ordered subunits. The transaction is then validated with at least one of the other nodes using the deterministic indicia. The validation indicates either a match or a mismatch of the deterministic indicia computed at the plurality of nodes.

Embodiments of the present application provide a database processing method, an electronic device, a storage medium, and a program product. The method comprises: receiving a data definition language (DDL) request sent by an application device, and executing each DDL statement in the DDL request on the basis of at least one first data node to obtain a first execution result of each DDL statement; if it is determined that the first execution result represents that the execution of the DDL statement has failed, updating an execution state of the DDL statement to a failed state, and determining the number of retries for the DDL statement; if it is determined that the number of retries for the DDL statement is less than or equal to a preset number, determining table information of a target database, and re-executing the DDL statement on the basis of the table information of the target database to obtain a second execution result of the DDL statement; and sending the second execution result of the DDL statement to the application device. The present application can improve database processing efficiency.

Systems and methods for generating a data definition language (DDL) statement for a database management product. The method includes receiving at an interface a partial DDL text input; providing the partial DDL text input to a machine learning model trained on a corpus of DDL statements and configured to analyze the received partial DDL text input and generate a complete DDL statement based on the analysis of the received partial DDL text input and the corpus of DDL statements.

The application provides a database master-slave synchronization method, device, equipment, medium and product, wherein the method comprises the following steps: obtaining a first redo log including a first globally unique number generated by a master database end for a first log as a data definition language log; in the case that a log analysis thread judges that the log type of the first redo log is the first occurrence, obtaining a corresponding first binary log cached in a relay log file according to the first globally unique number, and applying the binary log to a slave database through a database language thread. The database master-slave synchronization is realized based on the binary log and the redo log, and the method of generating the log based on the globally unique number is supplemented, so that the replication delay problem caused by the method of realizing the database master-slave synchronization based on the binary log only is solved, and the synchronization of the data definition language log is realized.

Systems and methods for generating a data definition language (DDL) statement for a database management product. The method includes receiving at an interface a partial DDL text input; providing the partial DDL text input to a machine learning model trained on a corpus of DDL statements and configured to analyze the received partial DDL text input and generate a complete DDL statement based on the analysis of the received partial DDL text input and the corpus of DDL statements.