Layered data processing method and system for rail transit

Abstract

Disclosed in the present invention are a layered data processing method and system for rail transit. The method comprises: on the basis of a service requirement, constructing data processing logic by means of a combination of SQL statements; and on the basis of a layering theory for a data warehouse, designing a data processing flow, and on the basis of the data processing logic, automatically executing the data processing flow, wherein the data processing flow comprises: acquiring raw data from a data source, and performing format processing; performing data quality inspection, data cleaning and data conversion on the raw data that has been subjected to the format processing; performing data aggregation and data statistics on the raw data that has been cleaned and converted, and generating an intermediate data table; and on the basis of the intermediate data table, performing data extraction and / or data analysis. In the present invention, the construction of data processing logic is simplified into the combination of SQL statements, thereby lowering the technical barrier and significantly improving the development and maintenance efficiency. The data processing flow is designed on the basis of the layering theory for the data warehouse, thereby realizing modular and layered management.

Description

A hierarchical data processing method and system for rail transit Technical Field

[0001] This invention relates to the field of rail transit data processing technology, and in particular to a hierarchical data processing method and system for rail transit. Background Technology

[0002] Rail transit generally refers to a transportation system where operating vehicles need to run on specific tracks. It is a complex system integrating multiple disciplines and trades, mainly composed of track routes, stations, rolling stock, maintenance and repair bases, power supply and substations, communication signals, and command and control centers. Common rail transit systems include traditional railways (ordinary railways, intercity railways, and suburban railways), subways, light rail, and trams.

[0003] Due to the diversified development of train and railway technologies, rail transit is exhibiting increasingly diverse types, as well as the inherent complexity of rail transit systems. These systems generate a wide variety and vast amounts of data, placing higher demands on the efficient management and processing of this data.

[0004] However, traditional data processing solutions typically rely on specialized programming languages ​​to implement data logic processing. The process of writing and debugging code is complex, with high technical barriers, hindering rapid iteration and functional expansion, and resulting in high development and maintenance costs. Furthermore, the data processing workflow in the rail transit industry is quite complex. When handling massive amounts of data, it often suffers from lengthy processing flows and unclear data flow, making it difficult to meet the industry's requirements for real-time performance and efficiency. Summary of the Invention

[0005] The purpose of this invention is to provide a hierarchical data processing method and system for rail transit, addressing the problems of existing data processing solutions relying too heavily on specialized programming languages ​​and having complex and lengthy data processing workflows. By simplifying the construction of data processing logic into a combination of SQL (Structured Query Language) statements, the technical threshold is lowered, and complex coding processes are avoided, significantly improving development and maintenance efficiency and helping to reduce costs. By designing the data processing workflow based on the hierarchical theory of data warehouses, complexity is reduced, modular and hierarchical management is achieved, and data processing efficiency is improved.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0007] The first aspect of this invention provides a hierarchical data processing method for rail transit, comprising: constructing data processing logic through a combination of SQL statements;

[0008] The data processing flow is designed based on the layered theory of data warehouses, and the data processing flow is automatically executed according to the data processing logic. The data processing flow includes:

[0009] Obtain raw data from the data source and perform format processing;

[0010] The original data after format processing is subjected to data quality checks, data cleaning, and data conversion.

[0011] The cleaned and transformed raw data are summarized and statistically analyzed to generate an intermediate data table;

[0012] Data extraction and / or data analysis are performed based on the intermediate data table.

[0013] Optionally, constructing the data processing logic through a combination of the SQL statements includes:

[0014] Based on business requirements, a preset SQL syntax template is selected, and the SQL statement is generated based on the SQL syntax template to construct the data processing logic.

[0015] Optionally, the SQL syntax template includes at least the following data processing operations: data cleaning, data summarization, data statistics, and data analysis.

[0016] Optionally, generating the SQL statement based on the SQL syntax template includes:

[0017] Directly call the SQL syntax template to generate the SQL statement; or

[0018] The SQL statement is generated by adjusting the SQL syntax template.

[0019] Optionally, constructing the data processing logic through a combination of the SQL statements includes:

[0020] During data processing, the SQL statements are dynamically generated based on business needs to construct and adjust the data processing logic.

[0021] Optionally, the configuration of the data processing flow can be adjusted according to business needs;

[0022] The configuration content includes at least: data source configuration, data processing logic configuration, task scheduling configuration, and multi-environment configuration.

[0023] Optionally, adjusting the configuration of the data source includes: adding, modifying, or deleting the data source; setting the connection information, data format, and data refresh frequency for each data source.

[0024] Optionally, adjusting the task scheduling configuration includes: performing timed scheduling, event-triggered scheduling, and batch scheduling of data processing tasks; and setting the execution frequency and priority of each data processing task according to business requirements.

[0025] Optionally, adjusting the multi-environment configuration includes:

[0026] Adjust the configuration information that needs to be modified according to different deployment environments;

[0027] The configuration information includes at least: database connection address, Kafka connection address, and file transfer size limit.

[0028] Optionally, the data source includes at least: track equipment data, train operation data, and passenger flow data.

[0029] A second aspect of the present invention provides a hierarchical data processing system for rail transit, comprising:

[0030] A logic generation module, used to construct data processing logic through combinations of SQL statements; and

[0031] A data processing module, which designs a data processing flow based on the hierarchical theory of data warehouses and automatically executes the data processing flow according to the data processing logic, the data processing module comprising:

[0032] The raw data layer is used to acquire and store raw data from the data source and to perform format processing on the raw data.

[0033] The detailed data layer is used to perform data quality checks, data cleaning, and data conversion on the original data after format processing.

[0034] A data aggregation layer, used to aggregate and statistically analyze the cleaned and transformed raw data, and generate intermediate data tables; and

[0035] An analysis data layer is used to extract and / or analyze data based on the intermediate data table.

[0036] Optionally, the logic generation module has a preset SQL syntax template;

[0037] The logic generation module is used to generate the SQL statement based on the SQL syntax template in order to construct the data processing logic.

[0038] Optionally, the logic generation module is used to dynamically generate the SQL statement during the data processing process in order to construct and adjust the data processing logic.

[0039] Optionally, the hierarchical data processing system further includes a configuration management module, which is used to adjust the configuration content of the data processing flow;

[0040] The configuration content includes at least: data source configuration, data processing logic configuration, task scheduling configuration, and multi-environment configuration.

[0041] Optionally, the data source includes at least: track equipment data, train operation data, and passenger flow data.

[0042] A third aspect of the present invention provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the method described in any one of the first aspects above.

[0043] A fourth aspect of the present invention provides a readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the method described in any one of the first aspects above.

[0044] This invention has at least the following technical effects:

[0045] By introducing SQL to construct data processing logic and replacing code writing with SQL, the construction of data processing logic is simplified to a combination of SQL statements. This not only lowers the technical threshold for data processing but also avoids complex code writing processes, significantly improving development and maintenance efficiency.

[0046] By applying the layered theory of data warehousing to the data processing scenarios of the rail transit industry, a layered architecture is constructed for the data processing flow in the rail transit industry. This optimizes the complex data processing flow in the rail transit industry and achieves modular and hierarchical management of data flow. Attached Figure Description

[0047] Figure 1 is a flowchart illustrating a hierarchical data processing method for rail transit according to an embodiment of the present invention.

[0048] Figure 2 is a schematic diagram of creating a data processing pipeline using SQL according to an embodiment of the present invention;

[0049] Figure 3 is a schematic diagram of data processing logic created via SQL according to an embodiment of the present invention;

[0050] Figure 4 is a schematic diagram of layering based on the layering theory of digital warehouses according to an embodiment of the present invention;

[0051] Figure 5 shows the engineering configuration service information provided in an embodiment of the present invention. Detailed Implementation

[0052] The following detailed description, in conjunction with the accompanying drawings and specific embodiments, provides a further detailed explanation of the hierarchical data processing method and system for rail transit proposed in this invention. The advantages and features of this invention will become clearer from the following description. It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions, used only to facilitate and clearly illustrate the embodiments of this invention. Please refer to the accompanying drawings to make the objectives, features, and advantages of this invention more apparent and understandable. It should be understood that the structures, proportions, sizes, etc., depicted in the accompanying drawings are only for illustrative purposes to aid those skilled in the art and are not intended to limit the implementation conditions of this invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in proportions, or adjustments to the size, without affecting the effects and objectives achieved by this invention, should still fall within the scope of the technical content disclosed in this invention.

[0053] To address the problems existing in current rail transit data processing solutions, this embodiment provides a hierarchical data processing method for rail transit, as shown in Figure 1. The method includes: constructing data processing logic through a combination of SQL statements; designing a data processing flow based on the hierarchical theory of data warehouses; and automatically executing the data processing flow according to the data processing logic. SQL, or Structured Query Language, is a database query and programming language used to access, query, update, and manage relational database systems. As a relatively simple and easy-to-understand query language, constructing data processing logic through the combination of SQL statements allows business personnel to complete data processing without programming skills, thus simplifying the data processing flow.

[0054] The aforementioned data warehouse generally refers to a system that centrally stores large amounts of data to support an organization's decision-making process. The layered theory of data warehouses is one of the foundations of data warehouse construction and design, aiming to store and manage data of different types and at different processing stages in a layered manner. As shown in Figure 1, based on the layered theory of data warehouses and combined with the characteristics of the rail transit industry, the data processing flow may include: obtaining raw data from the data source and performing format processing; performing data quality checks, data cleaning, and data transformation on the format-processed raw data; summarizing and statistically analyzing the cleaned and transformed raw data to generate intermediate data tables; and extracting and / or analyzing data based on the intermediate data tables. The data source includes at least: rail equipment data, train operation data, and passenger flow data.

[0055] To create the data processing logic using SQL, as shown in Figure 2, a data processing pipeline can first be created using SQL. Further, as shown in Figure 3, the data processing logic can be created using SQL. Specifically, to construct the data processing logic through a combination of SQL statements, the user can select a preset SQL syntax template according to business needs and generate SQL statements based on the template to construct the data processing logic. The SQL syntax template can include some common data processing operations in the data processing flow. Specifically, the SQL syntax template includes at least the following data processing operations: data cleaning, data summarization, data statistics, and data analysis.

[0056] To generate the SQL statement based on the SQL syntax template, the SQL statement can be generated either by directly calling the SQL syntax template or by modifying the SQL syntax template to meet specific business needs. Modifying the SQL syntax template allows for direct adjustments to the details of operations such as data cleaning, data aggregation, data statistics, and data analysis within the data processing flow, resulting in better adaptability.

[0057] Because rail transit systems generate a large amount of real-time data, this embodiment supports dynamically generated SQL statements to enhance the flexibility of SQL in data processing. That is, during data processing, SQL statements can be dynamically generated according to business needs to construct and adjust the data processing logic, and further automate its execution. The data processing flow can be automatically executed node by node according to user requirements, eliminating the need to write separate SQL statements for each processing node. In this way, the SQL logic can be automatically adjusted based on changes in actual data, thereby enabling real-time processing of dynamic data and significantly improving data processing efficiency and response speed. This enhances system flexibility while significantly improving the ability to process real-time rail transit data.

[0058] The following code is an example of dynamically generating the SQL statement:

[0059] Data is periodically stored in the data warehouse via the following interface:

[0060] public void executeSelect(){

[0061] List <datasourceentity>dataSourceEntities=dymDataSource.getDataSourcelist();

[0062] for (DataSourceEntity dataSourceEntity : dataSourceEntities){

[0063] if("I0TDB".equals(dataSourceEntity.getDatasource())){

[0064] List <deviceentity>devices = metaMangerService.selectDevice(dataSourceEntity.getPoolName());

[0065] for(DeviceEntity device :devices

[0066] ){

[0067] metaMangerService.updateDevice(dataSourceEntity.getPoolName(), device);

[0068] }

[0069] List <timeseriesentity>timeseriesEntities = metaMangerservice.selectTimeseries(dataSourceEntity.getPoolName());

[0070] for (TimeseriesEntity timeseries : timeseriesEntities){

[0071] metaMangerService.updateTimeseries(dataSourceEntity.getPoolName(),timeseries);

[0072] }

[0073] continue;

[0074] }

[0075] List <infoentity>infos = metaMangerService.selectInformation(dataSourceEntity.getPoolName());

[0076] List <colentity>colEntities = metaMangerService.selectColumn(dataSourceEntity,getPoolName());

[0077] for(InfoEntity info : infos){

[0078] metaMangerService.updateInformation(dataSourceEntity.getPoolName(), info);

[0079] }

[0080] for (ColEntity colEnty : colEntities){

[0081] metaMangerService.updateColumn(dataSourceEntity.getPoolName(), colEnty);

[0082] }

[0083] }

[0084] The code above allows you to iterate through the list of data sources using the executeSelect method and perform corresponding query and update operations based on the data source type.

[0085] Users can dynamically access platform data via SQL through the API:

[0086] public DataselectEntity selectByMySql(string sql, string dbType, Long pageNo, stringtable) {

[0087] DataSelectEntity dataSelectEntity = new DataSelectEntity();

[0088] Long offSet =(pageNo-1)* dataSelectEntity.getLimit();

[0089] List <Map<String, Object> >data = publishMapper.selectByMysql(sql, offset, dataselectEntity.getLimit());

[0090] dataSelectEntity.setData(data);

[0091] dataSelectEntity.setTotal(publishMapper.getTotalCount(table));

[0092] dataSelectEntity.setPage(pageNo);

[0093] return dataSelectEntity;

[0094] }

[0095] Using the code described above, this embodiment can dynamically parse the SQL statement passed in by the user, extract keywords, and return the content queried by the user.

[0096] In addition, this embodiment also supports the addition and deletion of dynamic multiple data sources:

[0097] public void autoLoad(){

[0098] List <datasourceentity>dbList = dymDataSource.getDataSourceList();

[0099] / / Traverse the data sources and load them

[0100] for(DataSourceEntity dataSourceEntity:dbList){

[0101] / / Decrypt

[0102] dataSourceEntity.setPassword(EncodeUtil.Base64Decoder(dataSourceEntity.getPassword()));

[0103] DataSourceProperty dataSourceProperty = new DataSourceProperty();

[0104] BeanUtils.copyProperties(dataSourceEntity, dataSourceProperty);

[0105] DynamicRoutingDataSource ds = (DynamicRoutingDataSource) dataSource;

[0106] DataSource dataSource = defaultDataSourceCreator.createDataSource(dataSourceProperty),

[0107] ds.addDataSource(dataSourceEntity.getPoolName(), dataSource);

[0108] L0GGER.info("name:{},type:{}", dataSourceEntity.getPoolName(), dataSourceEntity.getDatasource());

[0109] }

[0110] The above code allows you to use the autoLoad method to iterate through the list of data sources, decrypt each data source, and add it to the dynamic routing data source.

[0111] Given the complex and ever-changing business needs of the rail transit industry, traditional data processing systems often lack effective engineering configuration capabilities. Operations such as data source configuration and processing logic adjustments frequently require code modification, lacking a unified configuration management tool. Therefore, this embodiment further introduces an engineering configuration design to provide more flexible configuration management capabilities. Through engineering configuration, the configuration content of the data processing flow can be dynamically adjusted according to business needs to meet the requirements of different scenarios. The configuration content mainly includes data source configuration, data processing logic configuration, task scheduling configuration, and multi-environment system configuration.

[0112] Specifically, since rail transit systems often involve multiple data sources during data processing, adjusting the configuration of these data sources can include adding, modifying, or deleting them. Engineered configuration allows users to quickly add, modify, or delete data sources according to actual needs, facilitating flexible configuration of multiple data sources. Adjusting the data source configuration can also include setting parameters such as connection information, data format, and data refresh frequency for each data source, thereby achieving dynamic management of the data sources. Business personnel can dynamically add and delete required data sources according to their needs. Added data sources support modification of connection information, and various data source types can be selected, such as MySQL, Doris, and IoTDB.

[0113] In the data processing logic configuration, this embodiment allows users to directly adjust the data processing logic generated by SQL through configuration files or an interface. Users can define specific information data processing flows for different business scenarios, flexibly controlling the details of data cleaning, aggregation (summarization and statistics), and analysis. This allows for different processing logic to be implemented through configuration without modifying the code, enhancing the system's adaptability.

[0114] To adjust the task scheduling configuration, data processing tasks can be scheduled on a timed basis, triggered by events, or in batches. The execution frequency and priority of each data processing task can also be set according to business needs. Since rail transit data processing often requires timed or real-time task scheduling to meet business requirements, this embodiment provides flexible scheduling strategies for data processing tasks through engineering configuration, including timed scheduling, event-triggered scheduling, and batch scheduling. Users can set the execution frequency, priority, and other parameters of each data processing task according to actual needs to ensure that the system can process data efficiently.

[0115] To adapt to the needs of different deployment environments (such as development, testing, and production environments), this embodiment supports multi-environment configuration management. Adjusting the multi-environment configuration includes modifying the configuration information that needs to be modified for different deployment environments. Users can flexibly switch configurations in different deployment environments, ensuring that the system can quickly adapt to different business needs during deployment. Specifically, for example, the configurations that need to be modified in different environments can be exposed in a configmap. When deploying in different environments, the server's configuration information can be quickly modified by modifying the configmap, avoiding frequent image modifications. The configuration information includes, for example, database connection addresses, Kafka connection addresses, and file transfer size limits. The main functions of the configmap include storing configuration data, injecting configuration data, and dynamically updating configurations. By using the configmap, configuration information can be decoupled from the container image, allowing the configuration to be managed and updated independently of the application, thereby updating the configuration without rebuilding the image.

[0116] This embodiment addresses the practical needs of rail transit data processing by introducing an engineering-oriented configuration concept, supporting flexible configuration management functions such as multi-data source configuration, processing logic configuration, and task scheduling configuration. Through this configuration approach, the system can adapt to the diverse business requirements of the rail transit industry, not only improving the system's flexibility and scalability but also enhancing its reliability and ease of use when dealing with complex data processing scenarios. This configurable design allows the system to quickly adjust under different environments and requirements, improving its adaptability and scalability.

[0117] To implement the aforementioned hierarchical data processing method for rail transit, this embodiment provides a hierarchical data processing system for rail transit, including, for example, a logic generation module and a data processing module. The logic generation module is used to construct data processing logic through the combination of SQL statements, and the data processing module is used to design a data processing flow based on the hierarchical theory of data warehouses and automatically execute the data processing flow according to the data processing logic.

[0118] Based on the layered theory of data warehouses, as shown in Figure 4, the data processing module may include, for example, a raw data layer, a detailed data layer, a summary data layer, and an analytical data layer.

[0119] The raw data layer is used to acquire and store raw data from data sources and to perform format processing on the raw data. The raw data layer can connect to various subsystems of the rail transit system, such as the equipment management system, train operation monitoring system, and passenger flow management system. It acquires rail equipment data, train operation data, and passenger flow data from these systems as data sources, respectively. The data in the raw data layer undergoes simple format processing to ensure data integrity and consistency, providing a foundation for subsequent data processing.

[0120] The detailed data layer is used to perform data quality checks, data cleaning, and data transformation on the format-processed raw data. Specifically, data cleaning may include steps such as data deduplication, missing value imputation, and outlier handling. Through the processing of the detailed data layer, the reliability and accuracy of the data can be ensured, providing high-quality data input for the subsequent analysis data layer.

[0121] The summary data layer is used to summarize and statistically analyze the cleaned and transformed raw data, and generate intermediate data tables. Specifically, the summary data layer can employ a star schema to perform multi-dimensional summarization and statistics on the cleaned raw data, generating intermediate data tables that can be used by business personnel and the system. The main purpose of designing the summary data layer is to improve the efficiency of data query and processing, reduce direct access to unprocessed raw data, and avoid redundant calculations. By preprocessing and storing frequently used data in the summary data layer, the system's response speed can be effectively improved.

[0122] The analytical data layer, used for data extraction and / or analysis based on the intermediate data tables, serves as the final layer of the data processing architecture. This layer can be tailored to specific business needs, generating analytical results or providing decision support. Users can directly extract required data from the analytical data layer using SQL, or perform deeper analytical operations on it, providing support for operational decisions in the rail transit system. The analytical data layer offers several data analysis functions, such as historical data growth statistics and project distribution statistics. Business personnel can select different components from the data integration and data development modules to perform business data analysis based on their needs, such as alarm data statistics and data quality analysis.

[0123] The logic generation module has a pre-set SQL syntax template, which allows the generation of SQL statements based on the template to construct the data processing logic. The logic generation module can also dynamically generate the SQL statements during data processing to construct and adjust the data processing logic.

[0124] The hierarchical data processing system also includes a configuration management module, which is used to adjust the configuration content of the data processing flow. The configuration content includes at least data source configuration, data processing logic configuration, task scheduling configuration, and multi-environment configuration.

[0125] The hierarchical data processing system can be, for example, a big data platform that provides data processing interfaces. The front end of the big data platform may include a display module for human-computer interaction, and the back end may include the logic generation module and the data processing module. The display module can be used to display data processing logic, select and adjust SQL syntax templates, and input business requirements. Through the display module, the data processing logic can be displayed graphically, making it more intuitive and easier to modify and maintain.

[0126] Users can select corresponding functions in the display module according to their business needs. The backend automatically generates SQL statements and processes data based on these requirements, eliminating the need for manual SQL writing. This avoids cumbersome business processes and inconvenient modifications. By further pre-setting the SQL syntax templates, the threshold for data processing can be significantly lowered.

[0127] As shown in Figure 5, the display module may further include a configuration management interface, which provides workflow definition functionality to visualize the configuration management module. The aforementioned workflow may, for example, include a series of interconnected, automated data processing tasks.

[0128] Specifically, users can quickly add, modify, or delete the data sources according to their actual needs in the configuration management interface, and can also set the connection information, data format, and data refresh frequency of each data source.

[0129] Users can select configurations related to the data processing flow through different options on the configuration management interface, such as SQL type, data source, and task priority, thereby adjusting the data processing logic. Users can also directly adjust the data processing logic composed of SQL statements by entering a configuration file on the configuration management interface, avoiding direct code modification.

[0130] Users can create required workflow components and set workflow timings through the configuration management interface, controlling the workflow's online and offline times for scheduled execution. They can also set workflow trigger conditions for event-triggered scheduling. Furthermore, the configuration management interface allows for batch modification of user-configured workflows via engineering configuration, enabling batch scheduling. Users can set execution frequency, priority, and other parameters based on urgency to determine the sequential logic of different tasks.

[0131] Users can adjust the configuration information that needs modification for different environments through the configuration management interface. Specifically, the big data platform can expose the configuration information that needs to be modified for different deployment environments in the configmap. This allows for quick modification of server configuration information by modifying the configmap when deploying in different environments, avoiding frequent image modifications. The configuration information includes, for example, database connection addresses, Kafka connection addresses, and file transfer size limits.

[0132] In other aspects, this embodiment also provides an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, it implements the above-described hierarchical data processing method for rail transit.

[0133] In other aspects, this embodiment also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the above-described hierarchical data processing method for rail transit.

[0134] This invention introduces SQL to construct data processing logic, replacing code writing with SQL and simplifying the construction of data processing logic into a combination of SQL statements. This not only lowers the technical threshold for data processing but also avoids complex code writing, significantly improving development and maintenance efficiency. By applying the layered theory of data warehousing to the data processing scenarios of the rail transit industry, a layered architecture is implemented for the data processing flow in the rail transit industry. This optimizes the complex data processing flow in the rail transit industry, achieving modular and hierarchical management of data flow, representing a significant advancement compared to existing technologies.

[0135] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0136] It should be noted that the apparatus and methods disclosed in the embodiments herein can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments herein. In this regard, each block in a flowchart or block diagram may represent a module, program, or part of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system to perform the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0137] In addition, the functional modules in the various embodiments of this article can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0138] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.< / datasourceentity> < / colentity> < / infoentity> < / timeseriesentity> < / deviceentity> < / datasourceentity>

Claims

1. A hierarchical data processing method for rail transit, characterized in that, include: Data processing logic is constructed by combining SQL statements; The data processing flow is designed based on the layered theory of data warehouses, and the data processing flow is automatically executed according to the data processing logic. The data processing flow includes: Obtain raw data from the data source and perform format processing; The original data after format processing is subjected to data quality checks, data cleaning, and data conversion. The cleaned and transformed raw data are summarized and statistically analyzed to generate an intermediate data table; Data extraction and / or data analysis are performed based on the intermediate data table.

2. The hierarchical data processing method for rail transit according to claim 1, characterized in that, The process of constructing the data processing logic through a combination of SQL statements includes: Based on business requirements, a preset SQL syntax template is selected, and the SQL statement is generated based on the SQL syntax template to construct the data processing logic.

3. The hierarchical data processing method for rail transit according to claim 2, characterized in that, The SQL syntax template includes at least the following data processing operations: data cleaning, data summarization, data statistics, and data analysis.

4. The hierarchical data processing method for rail transit according to claim 2, characterized in that, The process of generating the SQL statement based on the SQL syntax template includes: Directly call the SQL syntax template to generate the SQL statement; or The SQL statement is generated by adjusting the SQL syntax template.

5. The hierarchical data processing method for rail transit according to claim 1, characterized in that, The process of constructing the data processing logic through a combination of SQL statements includes: During data processing, the SQL statements are dynamically generated based on business needs to construct and adjust the data processing logic.

6. The hierarchical data processing method for rail transit according to claim 1, characterized in that, The configuration of the data processing flow is adjusted according to business needs; The configuration content includes at least: data source configuration, data processing logic configuration, task scheduling configuration, and multi-environment configuration.

7. The hierarchical data processing method for rail transit according to claim 6, characterized in that, Adjustments to the data source configuration include: Add, modify, or delete the data source; Configure the connection information, data format, and data refresh frequency for each of the data sources.

8. The hierarchical data processing method for rail transit according to claim 6, characterized in that, Adjusting the task scheduling configuration includes: Perform timed scheduling, event-triggered scheduling, and batch scheduling for data processing tasks; Based on business requirements, set the execution frequency and priority of each data processing task.

9. The hierarchical data processing method for rail transit according to claim 6, characterized in that, Adjusting the multi-environment configuration includes: Adjust the configuration information that needs to be modified according to different deployment environments; The configuration information includes at least: database connection address, Kafka connection address, and file transfer size limit.

10. The hierarchical data processing method for rail transit according to claim 1, characterized in that, The data sources include at least: track equipment data, train operation data, and passenger flow data.

11. A hierarchical data processing system for rail transit, characterized in that, include: A logic generation module, which is used to construct data processing logic through the combination of SQL statements; and A data processing module, which designs a data processing flow based on the hierarchical theory of data warehouses and automatically executes the data processing flow according to the data processing logic, the data processing module comprising: The raw data layer is used to acquire and store raw data from the data source and to perform format processing on the raw data. The detailed data layer is used to perform data quality checks, data cleaning, and data conversion on the original data after format processing. A data aggregation layer, used to aggregate and statistically analyze the cleaned and transformed raw data, and generate intermediate data tables; and An analysis data layer is used to extract and / or analyze data based on the intermediate data table.

12. The hierarchical data processing system for rail transit according to claim 11, characterized in that, The logic generation module has a preset SQL syntax template; The logic generation module is used to generate the SQL statement based on the SQL syntax template in order to construct the data processing logic.

13. The hierarchical data processing system for rail transit according to claim 11, characterized in that, The logic generation module is used to dynamically generate the SQL statements during the data processing process in order to construct and adjust the data processing logic.

14. The hierarchical data processing system for rail transit according to claim 11, characterized in that, The hierarchical data processing system also includes a configuration management module, which is used to adjust the configuration content of the data processing flow; The configuration content includes at least: data source configuration, data processing logic configuration, task scheduling configuration, and multi-environment configuration.

15. The hierarchical data processing system for rail transit according to claim 11, characterized in that, The data sources include at least: track equipment data, train operation data, and passenger flow data.

16. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, implements the method of any one of claims 1 to 10.

17. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 10.

Images