Domain name data processing method and system, electronic device and storage medium
Patent Information
- Application Number
- PCT/CN2026/081209
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-04-16
- Filing Date
- 2026-03-03
- Publication Date
- 2026-10-22
Smart Images

Figure CN2026081209_22102026_PF_FP_ABST
Abstract
Description
Domain name data processing methods, systems, electronic devices, and storage media Technical Field
[0001] This disclosure relates to the Internet field, and more specifically, to a method, system, electronic device, and storage medium for processing domain name data. Background Technology
[0002] As a crucial component of network infrastructure, domain name servers face ever-increasing data volumes and processing complexity. In large-scale distributed domain name server environments, the collection, transmission, and storage of passive domain name resolution record data has become a significant technical challenge. Passive domain name resolution record data contains detailed information about domain name resolution queries and responses, which is of considerable value for network security monitoring, threat analysis, and brand protection.
[0003] However, due to the large volume of domain name resolution traffic, the transmission and storage of these passive domain name resolution record data in related technologies leads to high network bandwidth and storage costs, making it difficult to meet the needs of real-time and large-scale scenarios. In other words, the efficiency of collecting, transmitting and storing passive domain name resolution record data in related technologies is low and the cost is high.
[0004] There is currently no effective solution to the above problems. Summary of the Invention
[0005] This disclosure provides a method, system, electronic device, and storage medium for processing domain name data, to at least solve the technical problems of low efficiency and high cost in collecting, transmitting, and storing passive domain name resolution record data in related technologies.
[0006] According to one aspect of the present disclosure, a method for processing domain name data is provided, applied in a domain name server cluster, the domain name server cluster including at least one authoritative server and multiple recursive servers, the method comprising: acquiring domain name data from multiple recursive servers; aggregating the domain name data from multiple recursive servers according to the hierarchical structure of the multiple recursive servers to obtain at least one target aggregated data; and storing the at least one target aggregated data.
[0007] According to another aspect of the embodiments of this disclosure, a domain name data processing system is also provided, applied in a domain name server cluster, the domain name server cluster including at least one authoritative server and multiple recursive servers. The system includes: multiple data acquisition devices, respectively deployed in different recursive servers in the domain name server cluster, the data acquisition devices being configured to acquire domain name data of the corresponding recursive servers; a multi-level aggregation device, connected to the multiple data acquisition devices, configured to aggregate the domain name data of the multiple recursive servers according to the hierarchical structure of the multiple recursive servers to obtain at least one target aggregated data; and a data processing device, connected to the multi-level aggregation device, configured to store the at least one target aggregated data.
[0008] According to another aspect of the present disclosure, an electronic device is also provided, including: a memory storing an executable program; and a processor connected to the memory via a bus and configured to run the program, wherein the program executes the methods in various embodiments of the present disclosure when it runs.
[0009] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the methods of the various embodiments of the present disclosure.
[0010] According to another aspect of the embodiments of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this disclosure.
[0011] According to another aspect of the embodiments of this disclosure, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods in various embodiments of this disclosure.
[0012] According to another aspect of the embodiments of this disclosure, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this disclosure.
[0013] In this embodiment, the method is applied to a domain name server cluster containing at least one authoritative server and multiple recursive servers. First, domain name data from multiple recursive servers is acquired. Then, according to the hierarchical structure of the multiple recursive servers, this domain name data is aggregated to obtain at least one target aggregated data. Finally, this target aggregated data is stored. It is noteworthy that by collecting domain name data from multiple recursive servers, aggregating this data, and storing the resulting at least one target aggregated data, the data acquisition process avoids blind processing of all network traffic, significantly reducing the overall data size and thus lowering the costs of acquisition, transmission, and storage from the source. Aggregating and storing domain name data according to the hierarchical structure of the multiple recursive servers avoids transmitting large amounts of individual data separately, effectively reducing data transmission volume, reducing data processing overhead, and improving the efficiency of data transmission and storage. This enables the passive domain name resolution record system to handle large-scale network traffic, providing reliable technical support for subsequent data analysis, application decision-making, and security applications. This solves the technical problems of low efficiency and high cost in the acquisition, transmission, and storage of passive domain name resolution record data in related technologies.
[0014] It is worth noting that the above general description and the following detailed description are merely for illustrative and explanatory purposes and do not constitute a limitation thereof. Attached Figure Description
[0015] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:
[0016] Figure 1 is a schematic diagram of the hardware structure of a computer terminal according to an embodiment of the present disclosure;
[0017] Figure 2 is a schematic diagram of a computer terminal as a computing node in a computing environment according to an embodiment of the present disclosure;
[0018] Figure 3 is a schematic diagram of a structural block diagram of using a computer terminal as a service mesh according to an embodiment of the present disclosure;
[0019] Figure 4 is a flowchart of a domain name data processing method according to an embodiment of the present disclosure;
[0020] Figure 5 is a schematic diagram of a domain name data processing system according to an embodiment of the present disclosure;
[0021] Figure 6 is a schematic diagram of an optional domain name data processing system according to an embodiment of the present disclosure;
[0022] Figure 7 is a schematic diagram of an optional data compression using a buffer according to an embodiment of the present disclosure;
[0023] Figure 8 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation
[0024] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.
[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0026] First, some nouns or terms that appear in the description of the embodiments of this disclosure shall be interpreted as follows:
[0027] Domain Name System (DNS) is a system that provides a way to query the mapping relationship between domain names and Internet Protocol addresses.
[0028] An authoritative server is a server that stores resolution data identifying specific domain names and Internet Protocol addresses, and can provide query services for that data.
[0029] A recursive server typically does not store DNS resolution data; it only assists users in completing the domain name resolution process and returns the resolution results to the user. At the same time, it caches previous query results to improve the efficiency of subsequent queries.
[0030] Passive DNS is a passive DNS domain name resolution log collection system, generally used for security-related data analysis, hijacking monitoring, and other purposes.
[0031] According to embodiments of this disclosure, a method for processing domain name data is provided. It should be noted that the steps shown in the flowcharts in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0032] The method embodiments provided in this disclosure can be executed in a computer terminal or similar computing device. Figure 1 shows a hardware structure block diagram of a computer terminal for implementing a domain name data processing method. As shown in Figure 1, the computer terminal 10 may include one or more (shown as 102a, 102b, ..., 102n in the figure) processors 102 (processors 102 may include, but are not limited to, microprocessors MCUs or programmable logic devices FPGAs, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a cursor control device, a keyboard, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that the structure shown in Figure 1 is only illustrative and does not limit the structure of the above-described electronic device. For example, the computer terminal 10 may also include more or fewer components than shown in Figure 1, or have a different configuration than shown in Figure 1.
[0033] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuitry are generally referred to herein as "data processing circuitry". This data processing circuitry may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in embodiments of this disclosure, the data processing circuitry serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).
[0034] The memory 104 can be used to store software programs and modules of application software, such as program instructions and / or data storage devices corresponding to the methods in the embodiments of this disclosure. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the methods in the above embodiments. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0035] The transmission device 106 is used to receive or send data via a network and can establish wired and / or wireless network connections. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. The transmission device 106 may also include a network interface. In one example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.
[0036] The hardware structure block diagram shown in Figure 1 can serve as an exemplary block diagram not only for the computer terminal 10 described above, but also for the server described above. In an optional embodiment, Figure 2 illustrates an embodiment using the computer terminal 10 shown in Figure 1 as a computing node in the computing environment 201.
[0037] Figure 2 is a schematic diagram of a computer terminal as a computing node in a computing environment according to an embodiment of the present disclosure. As shown in Figure 2, the computing environment 201 includes multiple computing nodes (such as servers) running on a distributed network (shown as 210-1, 210-2, ... in the figure). Each computing node contains local processing and memory resources, and the terminal user 202 can remotely run applications or store data in the computing environment 201. Applications can be provided as multiple services 220-1, 220-2, 220-3, and 220-4 in the computing environment 201, representing services "A", "D", "E", and "H", respectively.
[0038] End user 202 can provide and access services through a web browser or other software application on a client. In some embodiments, the provisioning and / or requests of end user 202 can be provided to ingress gateway 230. Ingress gateway 230 may include a corresponding agent to handle the provisioning and / or requests for services (one or more services provided in computing environment 201).
[0039] Services are provided or deployed based on various virtualization technologies supported by the computing environment 201. In some embodiments, services may be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. VM-based virtualization can simulate a real computer by initializing a virtual machine, executing programs and applications without directly accessing any actual hardware resources. While the machine is virtualized by a virtual machine, container-based virtualization can launch containers to virtualize an entire operating system (OS), allowing multiple workloads to run on a single OS instance.
[0040] In one embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). For example, as shown in Figure 2, service 220-2 can be equipped with one or more Pods 240-1, 240-2, ..., 240-N (collectively referred to as Pods). A Pod can include a proxy 245 and one or more containers 242-1, 242-2, ..., 242-M (collectively referred to as containers). One or more containers in a Pod handle requests related to one or more corresponding functions of the service. The proxy 245 typically controls service-related network functions such as routing and load balancing. Other services can also be equipped with similar Pods.
[0041] During operation, executing a user request from end user 202 may require calling one or more services in computing environment 201. Executing one or more functions of one service requires calling one or more functions of another service. As shown in Figure 2, service "A" 220-1 receives a user request from end user 202 from ingress gateway 230. Service "A" 220-1 can call service "D" 220-2, and service "D" 220-2 can request service "E" 220-3 to execute one or more functions.
[0042] The aforementioned computing environment can be a cloud computing environment, where resource allocation is managed by cloud services, allowing functionality development without needing to consider implementation, adjustment, or server scaling. This computing environment allows developers to execute event-responsive code without building or maintaining complex infrastructure. Services can be partitioned into a set of functions that can automatically and independently scale, rather than scaling a single hardware device to handle potential loads.
[0043] In another alternative embodiment, FIG3 illustrates a block diagram of an example using the computer terminal 10 (or mobile device) shown in FIG1 above as a service mesh. FIG3 shows a structural block diagram of a service mesh 300, which is mainly used to facilitate secure and reliable communication between multiple microservices. Microservices refer to decomposing an application into multiple smaller services or instances and distributing them across different clusters / machines.
[0044] As shown in Figure 3, a microservice may include application service instance A and application service instance B, which together form the functional application layer of service mesh 300. In one implementation, application service instance A runs as a container / process 308 on machine / workload container group 314 (Pod), and application service instance B runs as a container / process 310 on machine / workload container group 316 (Pod).
[0045] In one implementation, application service instance A can be a product query service, and application service instance B can be a product order placement service.
[0046] As shown in Figure 3, application service instance A and grid proxy (sidecar) 303 coexist in machine / workload container group 314, and application service instance B and grid proxy 305 coexist in machine / workload container group 316. Grid proxy 303 and grid proxy 305 form the data plane layer of service mesh 300. Grid proxy 303 and grid proxy 305 run as containers / processes 304 and 306 respectively, and can receive requests 312 for product query services. Grid proxy 303 and application service instance A can communicate bidirectionally, and grid proxy 305 and application service instance B can also communicate bidirectionally. Furthermore, grid proxy 303 and grid proxy 305 can also communicate bidirectionally with each other.
[0047] In one implementation, traffic from application service instance A is routed to the appropriate destination via mesh proxy 303, and network traffic from application service instance B is routed to the appropriate destination via mesh proxy 305. It should be noted that the network traffic mentioned here includes, but is not limited to, Hypertext Transfer Protocol (HTTP), Representational State Transfer (REST), high-performance, general-purpose open-source frameworks (Google Remote Procedure Call, gRPC), and open-source in-memory data structure storage systems (Redis).
[0048] In one implementation, the functionality of the extended data plane layer can be achieved by writing custom filters for the proxy (Envoy) in service mesh 300. The service mesh proxy configuration can enable the service mesh to correctly proxy service traffic, achieving service interoperability and service governance. Mesh proxies 303 and 305 can be configured to perform at least one of the following functions: service discovery, health checking, routing, load balancing, authentication and authorization, and observability.
[0049] As shown in Figure 3, the service mesh 300 also includes a control plane layer. This control plane layer can consist of a set of services running in a dedicated namespace, managed by a managed control plane component 301 within machine / workload container groups (machine / Pods) 302. As shown in Figure 3, the managed control plane component 301 communicates bidirectionally with mesh agents 303 and 305. The managed control plane component 301 is configured to perform control and management functions. For example, it receives telemetry data from mesh agents 303 and 305 and can further aggregate this telemetry data. The managed control plane component 301 can also provide a user-facing Application Programming Interface (API) for these services, facilitating easier manipulation of network behavior and providing configuration data to mesh agents 303 and 305.
[0050] According to embodiments of this disclosure, a method for processing domain name data is provided. It should be noted that the steps shown in the flowcharts in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0051] In the above operating environment, this disclosure provides a method for processing domain name data as shown in Figure 4, which is applied to a domain name server cluster, the domain name server cluster containing at least one authoritative server and multiple recursive servers.
[0052] The aforementioned authoritative server can be a Domain Name System (DNS) authoritative server, responsible for storing and providing domain name resolution record information for a specific domain name. When a user enters a domain name in a browser, the operating system sends a query request to the DNS server to obtain the Internet Protocol (IP) address corresponding to that domain name. If the DNS server itself does not store the record information for that domain name, it will continue to send query requests to higher-level DNS servers until it finds an authoritative server containing the record information for that domain name. The authoritative server stores all the DNS record information for a specific domain name. When it receives a query request, the authoritative server can directly return the IP address or other record information corresponding to that domain name without sending further query requests to other servers. In other words, the authoritative server can act as the server that ultimately provides the domain name resolution result in the domain name resolution process.
[0053] The recursive server described above can be a server that requests queries from other domain name resolution servers and returns the final results. When a device needs to access a website or domain name, it first sends a query request to its local domain name resolution server. If the local domain name resolution server does not cache the Internet Protocol address of the domain name, it sends a query request to the root name server. The root name server tells the local domain name server which authoritative name server manages the domain name. The local domain name server then sends a query request to that authoritative name server again, and so on down the chain until it finds the domain name resolution server that manages the domain name. This process is the recursive domain name resolution query process. A recursive server can act as an intermediary server between a client device and other DNS servers on the Internet. When a client device queries a recursive DNS server for the Internet Protocol address corresponding to a domain name, the recursive server will recursively send query requests to other DNS servers until it finds the Internet Protocol address corresponding to the target domain name or encounters a query timeout. The recursive server can also cache query results to improve the speed of future identical queries. In other words, a recursive server can provide DNS resolution services to client devices, recursively query other DNS servers for the Internet Protocol address corresponding to a domain name, and cache query results to improve query speed.
[0054] Figure 4 is a flowchart of a domain name data processing method according to an embodiment of the present disclosure. As shown in Figure 4, the method may specifically include the following steps:
[0055] Step S402: Obtain domain name data for multiple recursive servers.
[0056] The domain name data mentioned above can be the interaction data between the recursive server and the authoritative server obtained by collection and filtering; that is, it can be passive domain name resolution record data.
[0057] In one alternative embodiment, domain name data from multiple recursive servers can be obtained. The network traffic of the recursive servers can be monitored in real time using a collection device. Detection technology can be used to filter only the domain name resolution request and response traffic sent from the recursive servers to the authoritative servers. This can greatly reduce unnecessary data processing and focus on the generation of Passive DNS data. Next, for the filtered domain name resolution response messages, the collection device can parse the domain name records, including but not limited to the ANSWER SECTION and ADDITIONAL SECTION, to extract key information such as the domain name, resolution type, resolution result, and the Internet Protocol address (SIP) of the recursive server and the Destination Internet Protocol address (DIP) of the authoritative server. The resolution type can include, but is not limited to, A records used to resolve domain names to their corresponding Internet Protocol version 4 (IPv4) addresses, MX records specifying mail server addresses, and NS records specifying name servers. The collected domain name resolution data can also be formatted into a uniform and easy-to-process format to facilitate efficient subsequent aggregation and analysis. This process also includes preliminary data cleaning to remove invalid or duplicate information.
[0058] In the above process, by collecting only the interaction data between the recursive server and the authoritative server, especially the response data from the authoritative server to the recursive server, blind processing of all network traffic is avoided, significantly reducing the overall data size and thus lowering storage and transmission costs. The intelligent filtering mechanism ensures that the collection device only processes relevant domain name resolution data, avoiding the processing of large amounts of irrelevant traffic and improving the overall processing performance and efficiency of the system. Simultaneously, the collection device ensures accurate capture of domain name resolution results, guaranteeing the integrity and accuracy of passive domain name resolution record data, providing a solid foundation for subsequent security analysis.
[0059] Step S404: According to the hierarchical structure of multiple recursive servers, aggregate the domain name data of multiple recursive servers to obtain at least one target aggregated data.
[0060] The aforementioned hierarchical structure refers to the level at which each recursive server performs its functions. The hierarchical structure can be determined based on factors such as the server room, data center, and geographical region where the recursive server is located. For example, recursive servers within the same server room can be considered to be at the same level. Alternatively, a data center may contain multiple server rooms, and the recursive server responsible for that data center can be considered to be the superior level of each recursive server in each server room. The specific hierarchical structure of the recursive server can be determined according to actual needs and is not limited here.
[0061] In one optional embodiment, domain name data of multiple recursive servers are aggregated according to their hierarchical structure. This aggregation can be based on geographical location and can be hierarchically structured. For example, the recursive servers can be divided into multiple geographical regions. Each geographical region can contain multiple data centers, each data center can contain multiple server rooms, and each server room can contain multiple recursive servers. This creates a tree-like structure with multiple recursive servers. An aggregation device responsible for a specific server room can aggregate the domain name data of each recursive server within that server room. An aggregation device for the data center above that server room can aggregate the data collected by the aggregation devices in each server room within that data center. Similarly, an aggregation device for the geographical region above that data center can aggregate the data collected by the aggregation devices in each data center within that geographical region. This allows for the aggregation of domain name data from multiple recursive servers according to their hierarchical structure. The number of hierarchical levels of the recursive servers can be determined based on actual needs; that is, the hierarchical data aggregation levels are not limited here. Alternatively, self-organizing aggregation based on network topology can be performed. Recursive servers automatically form aggregation chains based on their position in the network, such as tree, ring, or mesh structures. Then, each recursive server can act as an aggregation node, receiving and aggregating data from its direct neighbors, progressively passing the aggregation results to upper-level nodes until all data is aggregated to the central node. The aggregation path can be dynamically adjusted according to network conditions to achieve balanced data distribution and optimized transmission efficiency. Cloud-native elastic aggregation is also possible. Collection and aggregation services are deployed in the cloud, leveraging the elastic scaling characteristics of the cloud to dynamically adjust the number and distribution of aggregation nodes based on data traffic. The aggregation service can adopt a microservice architecture, with each microservice responsible for specific aggregation tasks, such as aggregation by domain name, type, or time window. Then, a cloud data stream processing platform can be used to achieve real-time aggregation and transmission of passive domain name resolution record data, while also supporting data replay and disaster recovery. The process of aggregating domain name data from multiple recursive servers to obtain at least one target aggregated data can be determined according to actual needs and is not limited here.
[0062] In the above process, by aggregating domain name data from multiple recursive servers, the amount of data transmission can be effectively reduced, alleviating the pressure on network bandwidth and data center storage, and lowering overall operating costs. Through data aggregation and compression, the overhead of data processing is reduced, and the efficiency of data transmission and storage is improved, enabling the passive domain name resolution record system to cope with large-scale network traffic. By optimizing the aggregation path and efficiency, it is ensured that passive domain name resolution record data can be transmitted to the data center quickly and in real time, providing timely data support for security analysis and threat detection.
[0063] Step S406: Store at least one target aggregated data.
[0064] In one optional embodiment, at least one target aggregated data is stored, which can be based on a relational database. After the aggregated data is compressed and integrated, it can be formatted into a relational database-compatible structure for easy storage. The target aggregated data can be periodically written to the database in batch processing, utilizing the database's transaction processing and indexing mechanisms to ensure data integrity and query efficiency. Master-slave replication, read-write separation, and load balancing technologies can be employed to ensure high availability of the storage system and avoid single points of failure. Alternatively, a distributed file system can be used for storage. The aggregated data is automatically sharded and allocated to different storage nodes according to a hash algorithm. Utilizing the characteristics of a distributed file system, it supports the storage and access of large-scale data. Each data shard is redundantly backed up on different storage nodes, improving data reliability and disaster recovery capabilities. The storage system can automatically adjust the number of nodes according to the increase or decrease in data volume, achieving elastic scaling of storage capacity to adapt to different application loads. It is also possible to use cloud storage-based storage methods, where the target aggregated data is stored in the form of objects in the cloud object storage service. By leveraging the high availability and low cost characteristics of the cloud object storage service, the data can be persistently stored. Alternatively, the stateless storage service of the cloud platform can be used to store the data in a storage pool, and each service node can read it on demand, thereby achieving data sharing and high-concurrency access.
[0065] In the above process, by storing at least one target aggregated data, the passive domain name resolution record system can effectively manage large-scale network data, ensure timely availability and long-term preservation of data, and achieve comprehensive improvement in cost control, performance optimization, high availability and scalability. This provides reliable data support for the passive domain name resolution record system in subsequent data analysis, application decision-making and security applications.
[0066] In this embodiment, the method is applied to a domain name server cluster containing at least one authoritative server and multiple recursive servers. First, domain name data from multiple recursive servers is acquired. Then, according to the hierarchical structure of the multiple recursive servers, this domain name data is aggregated to obtain at least one target aggregated data. Finally, this target aggregated data is stored. It is noteworthy that by collecting domain name data from multiple recursive servers, aggregating this data, and storing the resulting at least one target aggregated data, the data acquisition process avoids blind processing of all network traffic, significantly reducing the overall data size and thus lowering the costs of acquisition, transmission, and storage from the source. Aggregating and storing domain name data according to the hierarchical structure of the multiple recursive servers avoids transmitting large amounts of individual data separately, effectively reducing data transmission volume, reducing data processing overhead, and improving the efficiency of data transmission and storage. This enables the passive domain name resolution record system to handle large-scale network traffic, providing reliable technical support for subsequent data analysis, application decision-making, and security applications. This solves the technical problems of low efficiency and high cost in the acquisition, transmission, and storage of passive domain name resolution record data in related technologies.
[0067] In the above embodiments of this disclosure, the domain name data of multiple recursive servers are aggregated according to the hierarchical structure of multiple recursive servers to obtain at least one target aggregated data, including: aggregating the domain name data of recursive servers belonging to the same first-level node to obtain the initial aggregated data corresponding to the first-level node; and aggregating the initial aggregated data corresponding to the first-level node belonging to the same second-level node to obtain the target aggregated data.
[0068] In one optional embodiment, recursive servers can be divided into multiple hierarchical structures based on geographical regions, server processing levels, or logical regions to form multiple hierarchical nodes. The geographical regions can be different cities, different regional divisions, etc.; the server processing levels can refer to the rate at which the recursive servers generate domain name data; and the logical regions can be pre-defined management areas for the recursive servers. Within each first-level node, a data acquisition device can be deployed to capture domain name resolution request and response data between the recursive server and the authoritative server. After preliminary processing, including deblocking, extracting key information, and formatting, structured raw passive domain name resolution record data records are formed. Then, within each first-level node, the collected raw passive domain name resolution record data records can be transmitted to higher-level aggregation devices. These higher-level aggregation devices can be located at the network center of the region, responsible for receiving and initially aggregating data from all recursive servers within the region. The aggregation operation is performed according to domain name, domain name resolution record type, and resolution result, while simultaneously recording the source recursive server's Internet Protocol address, the authoritative server's Internet Protocol address, and the timestamp. The higher-level aggregation devices can also compress the data to reduce storage and transmission overhead. During the data aggregation process from the first-level nodes to the second-level nodes, the timeliness, integrity, and storage efficiency of the data can be considered. The aggregation device can adjust the aggregation strategy according to actual needs, such as merging duplicate records, optimizing data compression algorithms, and setting data retention periods, to adapt to larger-scale data processing requirements. Finally, the target aggregated data can be stored in a central data platform. This platform can use a distributed storage system to support efficient storage and access to massive amounts of data. The central data platform can also include operation and maintenance management functions for configuring rule distribution, status monitoring, and fault recovery to ensure stable system operation.
[0069] In the above process, layered aggregation reduces the amount of data transmitted from distributed collection points to the central data platform, thereby lowering network bandwidth and data center storage costs. Multi-level aggregation and data compression reduce the complexity of data processing and improve data transmission efficiency, enabling the system to efficiently process passive domain name resolution record data in large-scale network environments. At the same time, the multi-level aggregation architecture facilitates system expansion and can adapt to the ever-growing network scale and data volume. By performing multi-level aggregation of passive domain name resolution record data according to the hierarchical structure of recursive servers, the challenges of data transmission in large-scale network environments are effectively solved, providing an efficient and reliable passive domain name resolution record data management solution.
[0070] In the above embodiments of this disclosure, the domain name data of recursive servers belonging to the same first-level node are aggregated to obtain multiple initial aggregated data, including: for any recursive server belonging to the first-level node, reading at least one historical domain name data stored in a cache, wherein the cache is used to store the domain name data of the recursive server that has been aggregated within a preset period; if there is target domain name data in at least one historical domain name data that matches the domain name data of the recursive server, merging the domain name data of the recursive server with the target domain name data; if there is no target domain name data in at least one historical domain name data, storing the domain name data of the recursive server in the cache; and after all the domain name data of the recursive server has been processed, using at least one domain name data stored in the cache as the initial aggregated data.
[0071] The aforementioned preset period can refer to a time period pre-set for the cache, such as 30 minutes. The preset period can be determined according to actual needs and is not limited here.
[0072] In one optional embodiment, each recursive server can have a cache to store domain name data that has been aggregated within a preset period. The cache allows aggregation of domain name data from recursive servers belonging to the same first-level node. After the domain name resolution response message from the recursive server is intercepted and processed by the collection device, the resulting domain name data is sent to the primary aggregation device. When the primary aggregation device receives this data, it can check if there is target domain name data in the cache that matches the newly received domain name data. The matching criteria can include, but are not limited to, key fields such as domain name, query type, and response result. If a matching target domain name data is found in the cache, meaning the domain name's resolution record has already been collected, the aggregation device can merge the newly received domain name data with the target domain name data in the cache. The merging operation can include updating the source information of the response result and updating the start and end timestamps to reflect the recent active time range of this domain name record. For example, if the cache already contains the record example.com+A+1.1.1.1, and records its earlier and more recent occurrence times, when the same record is received again, the aggregation device will update its more recent occurrence time. If the target domain name data is not present in the cache, meaning the newly received domain name data is appearing for the first time, the aggregation device can store this data in the cache, waiting for subsequent potential matches or reaching a preset period. Once all domain name data from the recursive server—that is, data within a collection period—has been processed, and all records in the cache have been checked and possibly merged, the data in the cache will be used as initial aggregation data, ready for the next aggregation or reporting operation. At this stage, the data in the cache can be considered relatively new, deduplicated, and contains information within a time range.
[0073] Through the above process, data transmission overhead and storage requirements can be significantly reduced. It eliminates the need to transmit all DNS resolution responses completely to the data center during passive DNS record collection, avoiding large data volumes in large-scale networks. By employing a hierarchical aggregation approach, data on each recursive server undergoes initial filtering and merging at the primary aggregation stage. Only records that appear for the first time or are updated within a preset period are transmitted, significantly reducing the transmission of duplicate data and lowering data center storage pressure and network bandwidth consumption. Preliminary aggregation in the recursive server's cache can reflect changes in the network environment in real time, such as the appearance of new DNS records or changes to old records. This helps in real-time monitoring of network threats and optimizing DNS resolution service response time. Furthermore, dynamically updating timestamps allows for more accurate tracking of DNS resolution activity cycles, providing a foundation for time-series-based passive DNS record data analysis.
[0074] In the above embodiments of this disclosure, the historical key data in the target domain name data is the same as the key data in the domain name data of the recursive server. The historical key data consists of the historical query domain name, historical type and historical response result in the target domain name data, and the key data consists of the query domain name, type and response result in the domain name data of the recursive server.
[0075] In one optional embodiment, domain name data from the recursive server can generate a set of key data, which may include the query domain name, type, and response result. For example, a domain name data originates from server A, performs a type A query on example.com, and receives the IP response 1.1.1.1. The key data can be simplified to three key fields: example.com, type A, and 1.1.1.1. Historical key data can be stored in the cache of the first-level aggregation device, extracted from previously processed domain name data, and can also include historical query domain names, historical types, and historical response results. When new domain name data arrives at the aggregation device, the processor can compare the key data of the new domain name data with the historical key data in the cache to check if there are any identical key data combinations. The cache of the first-level aggregation device can construct an index structure based on the key data to accelerate the key data search and comparison process. When the processor finds a match, it can update the additional information in the corresponding historical key data, such as the most recent query timestamp and the recursive server address, to keep the key data synchronized with the new query status. If the processor fails to find a matching key in the cache, this can be considered a new query event. In this case, a new key entry can be created, and the complete domain name data information can be stored in the cache, ready for subsequent aggregation or transmission. Through this setup, key data comparison and updating can be achieved. The primary aggregation device can efficiently aggregate duplicate domain name resolution query data, avoiding redundant storage and significantly improving data processing speed and storage efficiency. It can also remove duplicate domain name data, retaining only the key data and additional information, reducing the amount of data and lowering data center storage costs.
[0076] In the above embodiments of this disclosure, merging the domain name data of the recursive server with the target domain name data includes: replacing the historical recursive server address in the target domain name data with the address of the recursive server, and replacing the historical authoritative server address in the target domain name server with the authoritative server address in the domain name data of the recursive server; storing the collection time of the domain name data of the recursive server into the target domain name data.
[0077] In one optional embodiment, when the primary aggregation device receives domain name data sent by the recursive server, it can compare the key data of this data—that is, the query domain name, type, and response result—with the historical key data of the target domain name data stored in the cache. If a match is found, i.e., the key data is the same, it can be determined that the domain name data of the current recursive server and the historical data already existing in the cache involve the same query type and response result for the same domain name, and the aggregation operation will be triggered. The aggregation operation may specifically include the following steps: replacing the historical recursive server address and the historical authoritative server address to update the historical server address information in the target domain name data with the new server address information. Specifically, the historical recursive server address in the target domain name data can be replaced with the address of the current recursive server, and the historical authoritative server address in the target data can be replaced with the new authoritative server address in the current domain name data to reflect the new network query path and authoritative source information. For example, if the cache contains a record for example.com+A+1.1.1.1, where the historical recursive server address and historical authoritative server address are historical server addresses, then upon receiving a new message with the same key, the historical recursive server address and historical authoritative server address will be updated to the addresses of the corresponding servers in the new message. Furthermore, the collection time can be updated. Specifically, the aggregation device can store the collection time of the current recursive server's domain name data in the target domain name data, or update the end timestamp of the target data to reflect the more recent activity time of this data. Updating the time information helps in subsequent analysis of passive domain name resolution record data based on time series, such as identifying abnormal behavior and monitoring trends in domain name resolution.
[0078] In the above process, key data matching and aggregation avoids storing large amounts of duplicate data in the data center, reducing storage costs and data processing complexity. Dynamically updating timestamps can reflect the latest status of domain name resolution records, which helps to monitor network activity in real time and respond quickly to security threats. By updating recursive server addresses and authoritative server addresses, the passive domain name resolution record system can better adapt to changes in network architecture, such as the migration or addition of domain name resolution servers, ensuring the accuracy and timeliness of data. The above aggregation mechanism achieves effective aggregation and deduplication of passive domain name resolution record data through data key matching, server address updating and timestamp management, while maintaining the real-time nature of data and the system's adaptability to network changes.
[0079] In the above embodiments of this disclosure, the domain name data of multiple recursive servers are aggregated according to the hierarchical structure of multiple recursive servers to obtain at least one target aggregated data, including: obtaining data filtering conditions; filtering the domain name data of multiple recursive servers based on the data filtering conditions to obtain filtered domain name data; and aggregating the filtered domain name data according to the hierarchical structure of multiple recursive servers to obtain at least one target aggregated data.
[0080] The aforementioned data filtering conditions can be filtering conditions issued to the aggregation device. Data filtering conditions can include, but are not limited to, specific domain names, query types, response results, or specific time periods. Data filtering conditions can be dynamically issued to each aggregation device by the data center's operation and maintenance management module according to application requirements and security policies to ensure that the collected data is consistent with the application objectives, while avoiding unnecessary data transmission and storage overhead.
[0081] In one alternative embodiment, data filtering conditions can be obtained and used to filter domain name data. When the primary aggregation device receives domain name data from the recursive server, it can apply the filtering conditions. For example, if the filtering condition excludes all data related to a malicious domain name list, then packets containing these malicious domain names will be ignored and will not be further processed or stored. Next, aggregation can be performed according to the hierarchical structure of the recursive servers. After completing data filtering, the primary aggregation device can aggregate the filtered domain name data. Aggregation considers data from multiple recursive servers at the same level or within the same region, reducing redundant data storage by merging data with the same key. For example, all A record query results for example.com received within the same region can be aggregated into a single record containing the addresses of different recursive servers, the address of the authoritative server, and timestamp information. The data aggregated by the primary aggregation device can then be sent to higher-level aggregation devices, such as intermediate aggregation devices. The multi-level aggregation process can be repeated until all data has undergone appropriate hierarchical aggregation and finally reaches the data platform in the data center. The data scope processed by each level of aggregation device can gradually expand from a local network to the entire region.
[0082] In the above process, by performing data filtering and preliminary aggregation at the primary aggregation device close to the data source, the amount of data transmitted to the data center can be significantly reduced, thereby lowering the data center's processing and storage costs. The multi-level aggregation mechanism enables real-time processing and aggregation of data, reducing data transmission latency and improving the real-time performance and availability of passive domain name resolution record data. Dynamically issued data filtering conditions allow the system to flexibly adjust data collection strategies according to different application scenarios and security requirements. At the same time, the multi-level aggregation structure can adapt to changes in network scale, effectively reducing data transmission and storage costs, and also improving the system's data processing efficiency and real-time performance.
[0083] In the above embodiments of this disclosure, obtaining domain name data of multiple recursive servers includes: for any one recursive server, filtering at least one response message between the recursive server and different authoritative servers; determining a target response message containing a preset status code from the at least one response message; and parsing the target response message to obtain domain name data.
[0084] In one optional embodiment, for each recursive server, the deployed collection device can continuously monitor and filter response messages between different authoritative servers. Filtering can be based on the attributes of the domain name resolution protocol messages, such as the source and destination Internet Protocol addresses, the type of domain name resolution response, and the status code of the response message. Considering that domain name resolution messages contain various status codes, such as NOERROR and SERVFAIL, the collection device can filter messages containing preset status codes to ensure data integrity and accuracy. For example, the NOERROR status code indicates a successful request and can be used as the type of data collected by the collection device, enabling passive domain name resolution record data to be collected based on successful domain name resolution records. Identifying target response messages containing preset status codes, such as NOERROR, from the filtered response messages ensures that the collected data is valid, i.e., the domain name query was successful and a response from the authoritative server was returned, thereby avoiding the collection and transmission of invalid or erroneous data and improving the overall data quality of the system. For a given target response message, the collection device can parse it and extract the domain name data, which may include the domain name contained in the parsing message, the domain name resolution query type, the resolution result returned by the authoritative server, and the response timestamp. The parsing process can follow the message format and response rules of the domain name resolution protocol to ensure that the extracted data is accurate and complete.
[0085] The above process effectively filters out invalid or erroneous interaction data between the recursive server and different authoritative servers, avoiding the consumption of storage space and transmission bandwidth by such data, and improving the efficiency and quality of data collection. The use of preset status codes ensures that successful domain name resolution records are collected. The resolution and extraction process of domain name data ensures that the collected data format is uniform and the information is complete, providing standardized data input for subsequent hierarchical aggregation and data processing in the data center, simplifying the data processing flow and improving processing efficiency.
[0086] In the above embodiments of this disclosure, parsing the target response message to obtain domain name data includes: parsing the target response message to obtain target message data in the target response message, wherein the target message data includes: the query domain name, type, and the response result corresponding to the query domain name; and generating domain name data based on the target message data, the address of the recursive server, and the address of the authoritative server.
[0087] In one optional embodiment, the data collection device parses the received target response message based on the message structure of the Domain Name Resolution Protocol (DNS). The DNS message consists of multiple parts, including a header, a Question Section, an Answer Section, an Authority Section, and an Additional Section. The data collection device can primarily focus on the Answer Section and the Additional Section, extracting the queried domain name, type, and corresponding response result. For example, from the Answer Section of the DNS message, the data collection device can read the A record 1.1.1.1 for example.com, and from the Additional Section, it can read the authority server information ns.example.com. After parsing the target message data, the data collection device can extract the recursive server address and the authority server address from the network layer message information. This information can be contained in the Internet Protocol Address (IPA) header. By reading the source IPA and target IPA fields, the data collection device can accurately record the recursive and authority server information related to the DNS response message. Finally, based on the parsed and extracted information described above, the data collection device can generate domain name data containing the queried domain name, type, response result, recursion server address, and authoritative server address. For example, for a successful example.com A record query, the domain name data generated by the data collection device could be as follows: {“query_domain”:“example.com”,“query_type”:“A”,“response_results”:“1.1.1.1”,“recursion_server_ip”:“SIP1”,“authoritative_server_ip”:“DIP1”}. This data contains key information about the domain name resolution query and also includes network layer server address information, providing the necessary context for further data aggregation and analysis.
[0088] In the above process, by accurately parsing the target response message, the acquisition device can accurately capture key information of the domain name resolution query, including the address information of recursive and authoritative servers. This is helpful for subsequent security analysis and decision-making based on passive domain name resolution record data. The acquired domain name data contains the address information of recursive and authoritative servers, which helps to provide network positioning capabilities for subsequent analysis, track the source of data, and respond to and investigate network security incidents. Parsing the target response message and generating structured domain name data can ensure the efficient and accurate operation of the passive domain name resolution record data acquisition and transmission method, improve the efficiency and quality of data acquisition, and provide a solid data foundation for subsequent data aggregation, storage, and analysis.
[0089] According to an embodiment of this disclosure, a domain name data processing system is also provided, which is applied in a domain name server cluster. The domain name server cluster includes at least one authoritative server and multiple recursive servers. Figure 5 is a schematic diagram of a domain name data processing system according to an embodiment of this disclosure. As shown in Figure 5, the system includes: multiple data acquisition devices 502, a multi-level aggregation device 504, and a data processing device 506.
[0090] The system includes multiple data acquisition devices 502, which are deployed in different recursive servers within a domain name server cluster. Each data acquisition device is configured to acquire domain name data from the corresponding recursive server. A multi-level aggregation device 504 is connected to the multiple data acquisition devices and is configured to aggregate the domain name data of the multiple recursive servers according to their hierarchical structure to obtain at least one target aggregated data. A data processing device 506 is connected to the multi-level aggregation device and is configured to store the at least one target aggregated data.
[0091] In one optional embodiment, a data acquisition device can be deployed on each recursive server in the domain name server cluster to monitor and collect domain name resolution response messages between the recursive server and the authoritative server in real time. The data acquisition device can identify and extract key information from the messages, including the queried domain name, query type, response result, and related timestamps. Considering that a recursive server can interact with multiple authoritative servers, the data acquisition device can process and filter response data from different authoritative servers to ensure data integrity and accuracy. The system can adopt a multi-level aggregation design. The primary aggregation device can be directly connected to the data acquisition device, responsible for initially aggregating data from recursive servers within the same region or cluster. This aggregation can be based on data entries with the same key, reducing data volume by merging duplicate entries. For example, query results for the same record A from example.com collected from different recursive servers will be merged into one record. The intermediate aggregation device further receives and aggregates data from multiple primary aggregation devices. Higher-level multi-level aggregation devices can also be set up. The aggregation level can be determined according to actual needs and is not limited here. This hierarchical aggregation method can effectively reduce the amount of data transmitted to the data center and reduce the storage pressure on the data center. Finally, the aggregated data, after undergoing multi-level aggregation, can be transmitted to the data processing unit in the data center for storage and further analysis. The data processing unit is responsible for persistent data storage and provides data query and analysis interfaces, supporting real-time application needs and offline data analysis. It can partition data storage according to application requirements, improving read / write efficiency, and reducing storage costs through data compression technology.
[0092] The above setup ensures real-time collection and rapid aggregation of passive DNS record data, enabling the data center to promptly acquire updated data and improving the real-time performance and accuracy of subsequent data analysis. Simultaneously, precise filtering by the collection device and efficient merging by the aggregation device guarantee high data quality and reduce interference from erroneous and invalid data. The multi-level aggregation design reduces redundant data transmission and alleviates storage pressure in the data center, significantly lowering system operating costs. Preliminary aggregation near the recursive server minimizes the amount of data transmitted to the data center, avoiding wasted network bandwidth, reducing the costs of passive DNS record data collection, transmission, and storage, improving the real-time performance and accuracy of data processing, and enhancing the system's fault tolerance and operational management flexibility.
[0093] In the above embodiments of this disclosure, the multi-level aggregation device includes: multiple primary aggregation devices deployed in different first-level nodes, wherein the primary aggregation devices are configured to aggregate domain name data of recursive servers belonging to the first-level nodes to obtain initial aggregated data; and at least one secondary aggregation device deployed in different second-level nodes, wherein the secondary aggregation devices are connected to the primary aggregation devices deployed in the first-level nodes belonging to the second-level nodes, and the secondary aggregation devices are configured to aggregate the initial aggregated data corresponding to the first-level nodes belonging to the second-level nodes to obtain target aggregated data.
[0094] In one optional embodiment, the system can employ a multi-level aggregation design. Specifically, the primary aggregation device deployed in each first-level node can be directly connected to the data acquisition device, responsible for initially aggregating data from recursive servers within the same first-level node. This aggregation can be based on data entries with the same key, reducing data volume by merging duplicate entries. For example, query results for the same record A from example.com collected from different recursive servers will be merged into a single record. The secondary aggregation device deployed in each second-level node can further receive and aggregate data from multiple primary aggregation devices. This hierarchical aggregation approach effectively reduces the amount of data transmitted to the data center and lowers the storage pressure on the data center.
[0095] In the above embodiments of this disclosure, a first-level aggregation device includes: a cache configured to store domain name data of any recursive server in a first-level node that has been aggregated within a preset period; a processor connected to the cache, configured to merge the domain name data of the recursive server with the target domain name data if, in at least one historical domain name data, there is target domain name data that matches the domain name data of the recursive server; store the domain name data of the recursive server in the cache if, in at least one historical domain name data, there is no target domain name data; and, after all the domain name data of the recursive server has been processed, use at least one domain name data stored in the cache as initial aggregation data.
[0096] In one optional embodiment, the cache can be used to temporarily store domain name data from recursive servers in the first-level nodes that has been aggregated within a preset period. This preset period can be dynamically adjusted based on system load and data update frequency to balance data real-time performance and storage efficiency. After the processor completes processing the domain name data from the recursive servers, if the data does not match historical data, this data can be stored in the cache, awaiting subsequent aggregation or transmission. The processor can collaborate with the cache to handle data aggregation, checking at least one historical domain name in the cache to determine if a target domain name already exists that matches the domain name data reported by the recursive servers. If a match is found, the processor will perform a data merging operation, updating the existing target domain name data, including adding or updating information such as the recursive server address, authoritative server address, and timestamp. If the processor does not find matching historical data, the domain name data from the recursive servers can be directly stored in the cache as a potential aggregation object. After all the domain name data from the recursive servers has been processed by the processor and stored in the cache, the first-level aggregation device performs final trimming and compression on at least one domain name data stored in the cache to generate initial aggregated data. The resulting data is then transmitted to a higher-level aggregation device or sent directly to the data center for further data storage and analysis.
[0097] With the above setup, the cache stores pre-aggregated data, allowing the processor to quickly check and aggregate similar data. This avoids the data center repeatedly processing large amounts of similar data, significantly improving the overall efficiency of data processing. The use of the cache not only reduces the storage pressure on the data center but also optimizes storage management by storing pre-aggregated data, making data easier to manage and retrieve.
[0098] In the above embodiments of this disclosure, the primary polymerization device is further configured to monitor the status of the currently connected secondary polymerization device, and in the event that the status of the currently connected secondary polymerization device is abnormal, connect to other secondary polymerization devices among at least one secondary polymerization device, wherein the other secondary polymerization device is any secondary polymerization device other than the currently connected secondary polymerization device among at least one secondary polymerization device.
[0099] In one optional embodiment, the primary aggregator can continuously monitor the status of its connected secondary aggregators, which can be achieved through a periodic heartbeat mechanism. The heartbeat mechanism checks whether the other party is online and responding by sending heartbeat packets. Specifically, the primary aggregator can periodically send heartbeat requests to the currently connected secondary aggregators. If no response is received within a preset time, the secondary aggregator is determined to be in an abnormal state. When an abnormal state of the currently connected secondary aggregator is detected, the primary aggregator can quickly execute a disaster recovery switchover process. The primary aggregator can select another secondary aggregator in a normal state from at least one secondary aggregator to connect to, thereby ensuring uninterrupted data transmission. This process can modify the communication configuration, re-establish the data transmission channel with the backup secondary aggregator, and report subsequent domain name data to the new aggregation point.
[0100] In the above setup, a heartbeat mechanism monitors the status of the secondary aggregation device and quickly switches to other available devices in case of anomalies. The primary aggregation device ensures the continuity of data transmission, guaranteeing the continued normal operation of the passive domain name resolution record data collection and transmission system even in the event of a partial network failure. The disaster recovery switching strategy provides redundant backups for the system, avoiding data transmission failures caused by single points of failure, and enhancing the system's fault tolerance and stability. By implementing heartbeat monitoring and disaster recovery switching strategies, the continuity and stability of passive domain name resolution record data collection and transmission are ensured.
[0101] In the above embodiments of this disclosure, the data acquisition device is further configured to monitor the status of the currently connected primary aggregation device, and in the event that the status of the currently connected primary aggregation device is abnormal, connect to other primary aggregation devices among the plurality of primary aggregation devices, wherein the other primary aggregation devices are any one of the plurality of primary aggregation devices other than the currently connected primary aggregation device.
[0102] In one optional embodiment, when the data acquisition device detects that the currently connected primary aggregator is in an abnormal state, such as high network latency, no response, or confirmed failure, it can immediately perform a disaster recovery switch. During the switchover process, a new connection can be established with other, normally functioning primary aggregators. The data acquisition device can pre-store or quickly retrieve information about available primary aggregators, including their network addresses and current operating status, to ensure that it can quickly find and connect to the next available aggregator in the event of a failure. The disaster recovery switchover mechanism of the data acquisition device ensures that even if the currently connected primary aggregator fails, data transmission can seamlessly switch to a backup aggregator, thereby avoiding data transmission interruption and ensuring the continuity of the passive domain name resolution record data stream.
[0103] In the embodiments described above, the data processing device is further configured to send data filtering conditions to multiple data acquisition devices and a multi-level aggregation device.
[0104] In one optional embodiment, the data processing device can generate data filtering conditions based on application requirements, security policies, or system performance considerations. These filtering conditions may include specific domain names, Internet Protocol address ranges, and domain name resolution query types, guiding the data collection and aggregation process and filtering out irrelevant or low-value data. The data processing device can send the filtering conditions to all data collection devices and multi-level aggregation devices through a preset communication channel, and through the data center's operation and maintenance channels to ensure that the filtering conditions are quickly and accurately transmitted to all levels of the system. By issuing data filtering conditions, the passive domain name resolution record system can selectively collect and aggregate data, avoiding the transmission and storage of invalid data, significantly improving the quality of processed data, and reducing the storage and analysis burden on the data center. The data processing device can dynamically adjust the filtering conditions according to real-time needs, enabling the entire passive domain name resolution record system to quickly adapt to application changes or security threats, improving the system's flexibility and adaptability.
[0105] The technical solution proposed in this disclosure is described below with reference to an optional embodiment. This disclosure proposes a compressible streaming passive DNS record (Passive DNS) data collection and transmission method, and proposes a passive DNS record data collection and storage method based on compressible streaming transmission. It mainly addresses the cost problem of transmission and storage caused by excessive data traffic in large-scale collection networks for passive DNS records. By introducing a layered aggregation network design, data transmission and storage are achieved without losing data integrity. This provides a real-time data stream for subsequent data analysis and hijacking monitoring functions based on passive DNS records.
[0106] Passive DNS records (DNS) are data generated during the operation of a DNS service. This data can help security teams trace the history of malicious activity, identify potential threat sources and victims, and is also significant and valuable in hijacking detection and brand protection. However, for large-scale distributed DNS service operators, extracting and transmitting passive DNS record data quickly and efficiently from trillions of records becomes a challenging task. This disclosure proposes a hierarchical aggregation model to collect and transmit passive DNS record data, using streaming and compressible methods to reduce the load and overhead in the entire data processing chain.
[0107] The technical solution disclosed herein can be divided into several parts: a data collection device, deployed on a domain name resolution recursive server, which filters out only the interaction data between the domain name resolution recursive server and the domain name resolution authoritative server through traffic collection, especially the response data packets of the authoritative server, and decapsulates the response data packets to obtain the domain name resolution results; an aggregation device, which mainly obtains the domain name resolution data obtained from the collection device through a message queue and performs hierarchical aggregation, compressing relevant data to achieve message data trimming and reduce the burden on the central data platform; and a data platform, which ultimately stores the passive domain name resolution record data for consumption by relevant applications or for offline data analysis and processing. The data platform also has operation and maintenance management functions, enabling the distribution of configuration rules and status management and monitoring of the collection device and aggregation device.
[0108] Figure 6 is a schematic diagram of an optional domain name data processing system according to an embodiment of the present disclosure. As shown in Figure 6, the solid lines with arrows represent data channels. The collection devices set in each domain name resolution recursive server can collect the interaction data between the domain name resolution recursive server and the domain name resolution authoritative server to obtain domain name data. The domain name data can then be transmitted to the corresponding primary aggregation device. After aggregation, the primary aggregation device can transmit the data to the corresponding secondary aggregation device. After aggregation, the secondary aggregation device can transmit the data to the data platform. The data platform includes real-time application consumption, offline processing, and operation and maintenance. The dashed lines with arrows represent operation and maintenance channels. The operation and maintenance in the data platform can manage the collection devices and aggregation devices to achieve operation and maintenance management of the collection devices, primary aggregation devices, and secondary aggregation devices. The dotted lines with arrows represent disaster recovery channels. The figure shows a disaster recovery channel between one collection device and another primary aggregation device, and also shows a disaster recovery channel between one primary aggregation device and another secondary aggregation device.
[0109] The detailed design of the device, the data collection device, is used to filter response data packets between recursive servers and authoritative servers. This filtering can refer to real-time collection of interactive domain name resolution data from the traffic; it can extract response information from domain name resolution protocol response packets in the traffic and extract response fields. For example, on server A, the data collection device obtains a domain name resolution response packet returning a "NOERROR" status. "NOERROR" can be a standard domain name resolution protocol status code, representing no error, and includes the corresponding response fields and additional fields as follows:
[0110] ANSWER SECTION;
[0111] example.com.3600 IN NS ns.example.com;
[0112] ADDITIONAL SECTION;
[0113] The data collection program will automatically extract the information, focusing on standard DNS resolution protocol messages. It will collect data based on a resolution result with a NOERROR status code (RCODE) and a non-empty response field in the ANSWER SECTION section. It will then append the server A's address (SIP) and authoritative DNS address (DIP), indicating the data packet source.
[0114] example.com.,NS,ns.example.com.,SIP1,DIP;
[0115] ns.example.com,A,1.1.1.1|2.2.2.2|3.3.3.3,SIP1,DIP.
[0116] After message collection, it is transmitted to the aggregation device through a message channel. The aggregation device located closest to the collection device can be a primary aggregation device, deployed locally, and responsible for managing all traffic within the same cluster. The aggregation device and the recursive server do not need to be located together, and the aggregation typically manages the data reporting of multiple recursive nodes. The collection device and the upstream aggregation device ensure continuous communication through heartbeats, and automatically switch to other available aggregators when the heartbeat fails.
[0117] The aggregation unit, with its layered design, reduces the excessive data load caused by direct connections between the collection end and the data center. Passive DNS records possess certain aggregation characteristics, which can be used to complete aggregation tasks. The secondary aggregation unit manages the data reported by all recursively collected DNS records within its region or a subset of regions. The management and maintenance of this data are handled by the data center, which distributes configurations accordingly. Proximity deployment ensures short links, reducing the risk of collection ends being blocked from reporting. The primary aggregation unit manages the data sent by the secondary aggregation units via message queues and maintains a heartbeat with the secondary aggregation units for automatic failover in case of anomalies.
[0118] The data aggregation rule is that when a message arrives at the aggregation device, it is not directly reported. Instead, it enters a temporary buffer. The buffer storage method is generally based on domain name + type + response as the storage key, and the actual information entry as the value. For example: the buffer stores three entries. One entry, example.com+A+1.1.1.1, represents the query domain name of example.com, type A, and the response result is 1.1.1.1. This information was collected at SIP1, sourced from DIP1, with a start time of T1 and a collection time of T2. The buffer receives a new entry, example.com+A+1.1.1.1, originating from collection point SIP3, with authoritative domain name resolution of DIP2, and a collection time of T6. When this message arrives, it will be compressed. The compression principle disclosed herein is to merge entries with the same domain name and type. The compressed data becomes example.com+A+1.1.1.1, corresponding to the stored information SIP3 and DIP2, with the start timestamp changed to T1 and the end time to T6.
[0119] Figure 7 is a schematic diagram of an optional data compression using a buffer according to an embodiment of the present disclosure. As shown in Figure 7, the buffer stores three entries from top to bottom. The first entry, “example.com+A+1.1.1.1→SIP1,DIP1,T1,T2”, can be compressed with the newly received entry “example.com+A+1.1.1.1,SIP3,DIP2,T6” to obtain the converted, i.e., compressed entry “example.com+A+1.1.1.1→SIP3,DIP2,T1,T6”. The other two entries in the buffer, “example.com+A+2.2.2.2|3.3.3.3→SIP2,DIP2,T2,T3” and “yun.com+NS+ns1.yun.com|ns2.yun.com→SIP1,DIP3,T4,T5”, do not need to be compressed for the time being, so they can remain unchanged.
[0120] The aggregation period and timing can be determined based on the local system hardware configuration. For example, servers supporting more memory can set a larger maximum number of entries in the cache, and when this threshold is exceeded, batch reporting is performed. However, a larger value also means increased data reporting latency. Therefore, the cache creation time is also considered when designing the system; for example, if more than 30 minutes have passed, reporting is performed directly regardless of the number of stored entries.
[0121] The data center receives and stores data reported from various aggregators. The actual storage process is similar to the memory aggregation cache mentioned above, but it uses physical storage. The cache performs short-term aggregation before writing to disk, avoiding immediate disk writes. Data is written all at once after a period of aggregation or when a certain storage capacity is exceeded. It provides data to various applications, and the operations and maintenance equipment can be used to distribute application running status and configurations. Disaster recovery switching is generally handled automatically through heartbeat information between the aggregation and collection devices. Distribution is typically based on load levels. For system stability, the following can also be considered: the issuance of security rules to block reports from certain domains or domains. For example, when an attack occurs, it's necessary to ensure that data is automatically filtered out to prevent contamination or exceeding storage capacity limits. When some nodes are heavily used by applications, the corresponding collection and reporting devices can be automatically interrupted to ensure the stable operation of core applications.
[0122] This disclosure presents an optimized design method for passive DNS records in large-scale data collection scenarios to address the problem of excessive data collection and transmission costs. Considering scalability and stability factors, and ensuring the validity of service data generation, it introduces an aggregation-layered design principle to reduce operational overhead and load, effectively helping medium and large enterprises lower the implementation cost of passive DNS records. The layered aggregation design solves the problem of excessive data collection and transmission volume in passive DNS records, which leads to high data center load; the introduction of a rule-based distribution model reduces the introduction of invalid data during data collection; and the inclusion of an operation and maintenance and disaster recovery design model ensures stable service operation and avoids impact on other applications.
[0123] Embodiments of this disclosure can provide an electronic device. FIG8 is a structural block diagram of an electronic device according to an embodiment of this disclosure. As shown in FIG8, the electronic device may include: an input / output device 802; a memory 804; and a processor 806, wherein the processor 806 is connected to the input / output device 802 and the memory 804 via a bus 808.
[0124] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0125] The processor can invoke an executable program stored in memory via a transmission device to perform the following methods: acquiring domain name data of multiple recursive servers; aggregating the domain name data of multiple recursive servers according to the hierarchical structure of the multiple recursive servers to obtain at least one target aggregated data; storing the at least one target aggregated data; and performing the methods in the various embodiments of this disclosure.
[0126] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.
[0127] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure.
[0128] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause the processing unit to execute the methods in the various embodiments of this disclosure.
[0129] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0130] Embodiments of this disclosure also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store program code executed by the method provided in the above embodiments.
[0131] Optionally, in this embodiment, the storage medium may be located in a computing device.
[0132] Optionally, in this embodiment, the computer-readable storage medium is configured to store an executable program, which, when the executable program is running, controls the device where the computer-readable storage medium is located to execute the method described in any of the above embodiments.
[0133] Embodiments of this disclosure also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.
[0134] The aforementioned computer program products can refer to software programs that have been written, tested, and released, and can run on computers or other devices. Computer program products can include application programs, operating systems, utility software, etc., used to achieve specific functions or solve specific problems.
[0135] Embodiments of this disclosure also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which can be used to store a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.
[0136] The aforementioned non-volatile computer-readable storage medium can refer to a medium for storing data. Non-volatile computer-readable storage media can retain data without loss when power is off and can be used to store long-term data, such as operating systems, applications, and user files. Non-volatile storage media can include hard disk drives, solid-state drives, optical disks, and flash memory storage devices, etc.
[0137] Embodiments of this disclosure also provide a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the method provided in the above embodiments.
[0138] The aforementioned computer program can refer to a set of instructions used to tell the computer to perform specific tasks or operations. Computer programs can be written by programmers using specific programming languages and can include algorithms, data structures, logic, and control flow. Computer programs can be used for a variety of purposes, including application software, operating systems, etc.
[0139] In the above embodiments of this disclosure, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0140] In the several embodiments provided in this disclosure, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.
[0141] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0142] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0143] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.
[0144] The above description is only a preferred embodiment of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of this disclosure, and these improvements and modifications should also be considered within the scope of protection of this disclosure.
Claims
1. A method for processing domain name data, applied to a domain name server cluster, the domain name server cluster comprising at least one authoritative server and multiple recursive servers, the method comprising: Obtain the domain name data of the multiple recursive servers; According to the hierarchical structure of the multiple recursive servers, the domain name data of the multiple recursive servers are aggregated to obtain at least one target aggregated data. The at least one target aggregated data is stored.
2. The method according to claim 1, wherein, The step of aggregating the domain name data of the multiple recursive servers according to their hierarchical structure to obtain at least one target aggregated data includes: The domain name data of recursive servers belonging to the same first-level node are aggregated to obtain the initial aggregated data corresponding to the first-level node; The initial aggregated data corresponding to the first-level nodes belonging to the same second-level node are aggregated to obtain the target aggregated data.
3. The method according to claim 2, wherein, The aggregation of domain name data belonging to the same first-level node of recursive servers yields multiple initial aggregated data, including: For any recursive server belonging to the first-level node, at least one historical domain name data stored in the cache is read, wherein the cache is used to store the domain name data of the recursive server that has been aggregated within a preset period. If, in the at least one historical domain name data, there exists target domain name data that matches the domain name data of the recursive server, the domain name data of the recursive server and the target domain name data are merged. If the target domain name data is not present in the at least one historical domain name data, the domain name data of the recursive server is stored in the cache. Once all domain name data on the recursive server has been processed, at least one domain name data stored in the cache is used as the initial aggregated data.
4. The method according to claim 3, wherein, The historical key data in the target domain name data is the same as the key data in the domain name data of the recursive server. The historical key data consists of the historical query domain name, historical type and historical response result in the target domain name data, and the key data consists of the query domain name, type and response result in the domain name data of the recursive server.
5. The method according to claim 3, wherein, The step of merging the domain name data of the recursive server with the target domain name data includes: Replace the historical recursive server address in the target domain name data with the address of the recursive server, and replace the historical authoritative server address in the target domain name server with the authoritative server address in the domain name data of the recursive server. The collection time of the domain name data of the recursive server is stored in the target domain name data.
6. The method of any one of claims 1 to 5, wherein, The step of aggregating the domain name data of the multiple recursive servers according to their hierarchical structure to obtain at least one target aggregated data includes: Get data filtering conditions; Based on the data filtering conditions, the domain name data of the multiple recursive servers are filtered to obtain the filtered domain name data. According to the hierarchical structure of the multiple recursive servers, the filtered domain name data is aggregated to obtain at least one target aggregated data.
7. The method of any one of claims 1 to 5, wherein, The step of obtaining the domain name data of the plurality of recursive servers includes: For any given recursive server, filter at least one response message between the recursive server and different authoritative servers; From the at least one response message, determine the target response message containing the preset status code; The target response message is parsed to obtain the domain name data.
8. The method of claim 7, wherein, The step of parsing the target response message to obtain the domain name data includes: The target response message is parsed to obtain the target message data in the target response message, wherein the target message data includes: the query domain name, the type, and the response result corresponding to the query domain name; The domain name data is generated based on the target message data, the address of the recursive server, and the address of the authoritative server.
9. A domain name data processing system, applied in a domain name server cluster, the domain name server cluster comprising at least one authoritative server and multiple recursive servers, the system comprising: Multiple data acquisition devices are deployed in different recursive servers in a domain name server cluster. The data acquisition devices are configured to acquire domain name data of the corresponding recursive servers. A multi-level aggregation device, connected to the plurality of data acquisition devices, is configured to aggregate the domain name data of the plurality of recursive servers according to the hierarchical structure of the plurality of recursive servers, to obtain at least one target aggregated data. A data processing device, connected to the multi-level aggregation device, is configured to store the at least one target aggregated data.
10. The system of claim 9, wherein, The multi-stage polymerization apparatus includes: Multiple primary aggregation devices are deployed in different first-level nodes. The primary aggregation devices are configured to aggregate domain name data of recursive servers belonging to the first-level nodes to obtain initial aggregated data. At least one secondary aggregation device is deployed in different second-level nodes. The secondary aggregation device is connected to a primary aggregation device deployed in a first-level node belonging to the second-level node. The secondary aggregation device is configured to aggregate the initial aggregation data corresponding to the first-level node belonging to the second-level node to obtain the target aggregation data.
11. The system of claim 10, wherein, The primary polymerization apparatus includes: The cache is configured to store domain name data of any recursive server in a first-level node that has been aggregated within a preset period; The processor, connected to the cache, is configured to: merge the domain name data of the recursive server with the target domain name data if, in the at least one historical domain name data, there is target domain name data that matches the domain name data of the recursive server; store the domain name data of the recursive server in the cache if, in the at least one historical domain name data, there is no target domain name data; and, after all the domain name data of the recursive server has been processed, use at least one domain name data stored in the cache as the initial aggregated data.
12. The system of claim 10, wherein, The primary aggregator is also configured to monitor the status of the currently connected secondary aggregator, and if the status of the currently connected secondary aggregator is abnormal, connect to another secondary aggregator among the at least one secondary aggregator, wherein the other secondary aggregator is any secondary aggregator among the at least one secondary aggregator other than the currently connected secondary aggregator.
13. The system of any one of claims 9 to 12, wherein, The data acquisition device is also configured to monitor the status of the currently connected primary aggregation device, and if the status of the currently connected primary aggregation device is abnormal, connect to other primary aggregation devices among the plurality of primary aggregation devices, wherein the other primary aggregation devices are any one of the plurality of primary aggregation devices other than the currently connected primary aggregation device.
14. The system of any one of claims 9 to 12, wherein, The data processing device is also configured to send data filtering conditions to multiple data acquisition devices and the multi-level aggregation device.
15. An electronic device applied to a domain name server cluster, the domain name server cluster comprising at least one authoritative server and multiple recursive servers, the electronic device comprising: Memory, which stores executable programs; A processor is configured to run the program, wherein the program executes the following method during runtime: acquiring domain name data of the plurality of recursive servers; aggregating the domain name data of the plurality of recursive servers according to the hierarchical structure of the plurality of recursive servers to obtain at least one target aggregated data; The at least one target aggregated data is stored.
16. The electronic device of claim 15, wherein, The processor is further configured to perform the following method: aggregating domain name data of recursive servers belonging to the same first-level node to obtain initial aggregated data corresponding to the first-level node; aggregating the initial aggregated data corresponding to the first-level node belonging to the same second-level node to obtain the target aggregated data.
17. The electronic device of claim 16, wherein, The processor is further configured to perform the following method: for any recursive server belonging to the first-level node, read at least one historical domain name data stored in a cache, wherein the cache is used to store the domain name data of the recursive server that has been aggregated within a preset period; if there is target domain name data in the at least one historical domain name data that matches the domain name data of the recursive server, merge the domain name data of the recursive server with the target domain name data; if there is no target domain name data in the at least one historical domain name data, store the domain name data of the recursive server in the cache; and if the domain name data of the recursive server has been processed, use the at least one domain name data stored in the cache as the initial aggregation data.
18. The electronic device of claim 17, wherein, The historical key data in the target domain name data is the same as the key data in the domain name data of the recursive server. The historical key data consists of the historical query domain name, historical type and historical response result in the target domain name data, and the key data consists of the query domain name, type and response result in the domain name data of the recursive server.
19. The electronic device of claim 17, wherein, The processor is further configured to perform the following method: replacing the historical recursive server address in the target domain name data with the address of the recursive server, and replacing the historical authoritative server address in the target domain name server with the authoritative server address in the domain name data of the recursive server; The collection time of the domain name data of the recursive server is stored in the target domain name data.
20. A computer readable storage medium comprising a stored executable program, wherein, When the executable program is executed, it controls the device containing the storage medium to perform the method described in any one of claims 1 to 8.