Liquid immersion cooling system and data center equipped with the same

Abstract

The present invention reduces overall cost by adopting a circulation-type configuration using natural water (river water, sea water, dam water, etc.), such as groundwater, and improves the cooling efficiency, the operational safety, and the lifespan of equipment to be cooled. The present invention is a liquid immersion cooling system comprising: a liquid immersion tank 2 that accommodates equipment to be cooled, and that is filled with a highly insulating liquid agent; water tanks 4A, 4B that store water; and a heat exchanger 3 that exchanges heat between the water in the water tanks and a solvent. The water tanks store water supplied from a natural water source, allow the water to flow out to the heat exchanger via an injection mechanism, receive the heated water from the heat exchanger via a discharge mechanism, and discharge the water to the natural water source.

Description

Immersion cooling system and data center equipped with same

[0001] The present invention relates to a technology related to immersion cooling, which improves heat dissipation and reduces power consumption by immersing server equipment and the like in an insulating liquid.

[0002] Since the birth of the Internet in 1995, the spread of digital technology has completely transformed global connectivity. Furthermore, with the widespread penetration of communication technologies such as broadband, 4G, and 5G, various industries including communication, commercial transactions, and entertainment are undergoing transformation.

[0003] On the other hand, this digital revolution has progressed hand in hand with environmental concerns. Since the early 2010s, global movements aiming for carbon neutrality have become active, and efforts have been made to improve energy efficiency, but this progress tends to be offset by the dramatic increase in high-performance computing (HPC).

[0004] Furthermore, the evolution of generative AI, represented by ChatGPT launched in the second half of 2022, has accelerated by 2024, and demand for computing resources has further expanded. For example, the construction of new data centers and the increase in power consumption have become major issues in terms of sustainability.

[0005] In response to such circumstances, conventional cooling methods such as air cooling and water cooling have become difficult to meet the requirements for thermal management and energy efficiency of data centers where GPUs play a central role. Against this background, immersion cooling technology has attracted attention, and various proposals have been made for improving heat dissipation, reducing power consumption, and downsizing system design by submerging server equipment in insulating liquid.

[0006] For example, Patent Document 1 discloses a data center comprising: an immersion tank holding an information processing device in a coolant; a sealed water-spraying and forced-air cooling system in which the coolant from the immersion tank flows through the piping and water is sprayed onto the piping exposed to outside air taken in through an outside air intake to cool the piping; a first temperature measuring unit that measures a first temperature which is the temperature of the coolant from the cooling system; and a cooling system control unit that controls the operation of the water-spraying and forced-air cooling units by controlling an ON / OFF control signal of the cooling system based on the temperature measured by the first temperature measuring unit.

[0007] Patent No. 6980969

[0008] However, Patent Document 1 does not disclose anything about a circulating system that uses natural water such as groundwater.

[0009] This invention has been made in view of these problems, and its purpose is to reduce overall costs by using a circulating system that utilizes water from natural water sources such as groundwater (groundwater, river water, seawater, dam water, etc.), while also providing a technology that improves the cooling efficiency, operational safety, and lifespan of the equipment to be cooled.

[0010] To solve the above problems, an immersion cooling system according to one aspect of the present invention comprises an immersion tank containing the equipment to be cooled and filled with a highly insulating liquid agent, a first water tank for storing water from a natural water source, a heat exchanger for performing heat exchange between the water in the water tank and the liquid agent, and a second water tank for storing the water used in the heat exchanger. The first water tank discharges water supplied from the natural water source to the heat exchanger, and the second water tank receives the heated water from the heat exchanger, discharges a portion of it to the first water tank, and discharges the rest to the natural water source.

[0011] According to the present invention, by adopting a circulating system that uses water from natural water sources such as groundwater (groundwater, river water, seawater, dam water, etc.), it is possible to reduce overall costs and provide a technology that improves the cooling efficiency, operational safety, and lifespan of the equipment to be cooled.

[0012] Figure 1 is a diagram showing the configuration of an immersion cooling system according to an embodiment of the present invention. Figure 2 is an external view of the system. Figure 3 shows the configuration of a data center employing the system. Figure 4 is a diagram showing the configuration of the control system of the data center employing the system. Figures 5(a) and 5(b) show experimental results.

[0013] <Features of this system> With the rapid expansion of AI and IoT in recent years, the infrastructure load on data centers has increased dramatically. Hyperscale (large-scale, centralized) data centers, which have been the mainstream until now, require a long construction period of 3 to 5 years and have the problem of power supply being concentrated in one place. In addition, there are concerns that the response time required for services will be long and the impact on the environment is serious. In recent years, small-scale, distributed facilities called edge data centers have attracted attention. These edge data centers are particularly useful for applications where low response time is essential, such as autonomous vehicles, industrial IoT, and smart cities, and their container-type modular system, which is easy to introduce and expand, enhances operational flexibility and robustness. In this invention, we have further devised a way to realize a container-type edge data center by distributing small-scale data centers to multiple locations and combining them with distributed storage technology that includes data encryption, thereby enhancing overall security.

[0014] Furthermore, containerized edge data centers have the following characteristics:

[0015] (1) Adaptability to diverse installation configurations: Installation on small plots of land: Compact 10ft / 20ft / 40ft containers can be installed even on limited sites in urban areas or remote areas. Expansion of hyperscale facilities: Modules can be added to large-scale data centers to expand computing resources while minimizing environmental impact. On-premise deployment: For companies that require confidentiality and privacy, a configuration that allows them to operate their own servers is available.

[0016] (2) Sustainable cooling system centered on liquid immersion cooling Excellent thermal performance: High heat dissipation is achieved by immersing electronic components in an insulating liquid. Improved energy efficiency: Significant power savings are achieved by reducing the need for large air conditioning units. Compact design: Installation space can be reduced as the cooling equipment is simplified.

[0017] (3) Scalability and adaptability Power supply capacity of up to 0.5 MW: Capable of handling high-performance GPU workloads. Modular expansion: Achieves high scalability while shortening construction time compared to conventional data centers.

[0018] (4) Integration with renewable energy Combination with solar and wind power: Designed to easily connect to renewable energy sources in order to further enhance sustainability.

[0019] Thus, the containerized edge data center according to the embodiment of the present invention can be described as a next-generation approach to building digital infrastructure that combines high performance, flexibility, and sustainability. Not only is it easy to deploy regardless of location or scale, but when combined with advanced cooling technology, it enables a wide range of entities, including businesses, public organizations, and hyperscalers, to rapidly deploy energy-efficient data centers. The multi-purpose use of small-scale edge data centers could pave the way for a new era of distributed computing.

[0020] <Embodiments> Embodiments of the present invention will be described in detail below.

[0021] Figure 1 shows a conceptual diagram of a liquid immersion cooling system according to one embodiment of the present invention, Figure 2 shows the external configuration of the system, and Figure 3 shows a data center employing the liquid immersion cooling system.

[0022] As shown in these figures, the immersion cooling system 1 according to an embodiment of the present invention includes an immersion tank 2, a heat exchanger 3, a first water tank 4A, a second water tank 4B, and water supply pumps 6A and 6B. The immersion tank 2 and the heat exchanger 3 are housed in a rack 10, and the immersion tank 2 houses server equipment, network equipment, or externally connectable equipment containing GPUs or ASICs. The inside of the immersion tank 2 is filled with a highly insulating liquid. Note that for server equipment and the like, the fan mechanism may be removed when housing them in the immersion tank 2.

[0023] In this example, as shown in Figure 3, twelve racks 10 are housed in containers 200A, 200B, etc., and multiple containers 200A, 200B, etc. are installed inside the building 201. The first water tank 4A and the second water tank 4B are installed outdoors. A simple main building / warehouse can be used to accommodate heavy snowfall areas, and a simple roof can be used to accommodate normal areas. Furthermore, in this example, each container 200A, 200B, etc. is connected to a wet (water) utility tunnel 202 and a dry (network, electricity, surveillance cameras, security, etc.) utility tunnel 203, respectively. A UPS or battery may be installed separately if necessary.

[0024] The first water tank 4A receives water from natural water sources (e.g., groundwater, river water, seawater, dam water, etc.) or tap water (approximately 14 degrees Celsius, 480 L / min per container) pumped up by the water supply pump 6A via an inlet path, and constantly stores this water. The first water tank 4A also accepts the inflow of a portion (30% in this example) of the hot water (approximately 30 degrees Celsius) from the second water tank 4B. The water in the first water tank 4A (approximately 14-20 degrees Celsius, 480 L / min per container) is then supplied to the heat exchanger 3 by the water supply pump 6B. Each rack of the immersion cooling system 1 requires 40 L / min of water for cooling, and in this embodiment, it is assumed that 12 racks are housed in one container, so the first water tank 4A stores 480 L / min of water.

[0025] The immersion tank 2 is connected to a heat exchange system 14, including a heat exchanger 3, via an upper circulation section 12 or a middle circulation section 12. The liquid agent, warmed by the heat dissipation of the equipment contained in the immersion tank 2, is circulated between the immersion tank 2 and the heat exchanger 3 by the action of a circulation pump 15. In the heat exchanger 3, the liquid agent is cooled by cooling water (14-20 degrees Celsius, 40 L / min) and returned to the immersion tank 2. When the immersion tank 2 is filled to the brim with liquid agent, the circulation of the liquid agent is carried out by the upper circulation section 12. When the liquid agent is filled to about half its brim, the circulation of the liquid agent is carried out via the middle circulation section 12. Cooling water flows into the heat exchanger 3 from the inlet passage 13a, and warmed water is discharged from the outlet passage 13b. Filters may be installed in the circulation path between the immersion tank 2 and the heat exchanger 3 to remove dust, dirt, deteriorated liquid agent, etc. that may be mixed into the liquid agent. Based on this mechanism, it can be said that the rack in question is a closed-loop, single-phase, proprietary immersion system.

[0026] The hot water (approximately 30 degrees Celsius, 480 L / min per container) used for heat exchange in the heat exchanger 3 is discharged to the second water tank 4B via the discharge channel 13b. Of the approximately 30-degree water in the second water tank 4B, 30% (144 L / min per container) is discharged to the first water tank 4A, and 70% (336 L / min per container) is discharged to the natural water source 6 via the discharge channel 13b. In this way, by discharging 30% of the used hot water to the first water tank 4A, the temperature of the water flowing in from the natural water source 5 is warmed to approximately 14-20 degrees Celsius, thereby improving the efficiency of heat exchange by the heat exchanger 3.

[0027] This circulation system cools the various devices housed in the immersion tank 2 within the rack 10. Furthermore, 70% of the heated 30-degree water discharged to the second water tank 5 is cooled to some extent before being discharged to the natural water source 6 via the discharge passage 13b, thus achieving an environmentally friendly, circulating cooling system.

[0028] In other words, with the above configuration, by using water from natural water sources such as groundwater as the water used in the heat exchanger 3, it is possible to significantly reduce operating costs. Furthermore, by returning the extracted groundwater to the natural water source 5, the environmental impact can also be reduced. In addition to the water from the natural water source 5, it is also possible to use industrial wastewater as the water used in the heat exchanger 3, in which case it can be recycled. Alternatively, tap water can also be used as the water used in the heat exchanger 3, in which case it is expected that the water heated in the heat exchanger 3 can be used as a heat source for other purposes. In addition, it is also possible to use coolants other than water, such as alternative refrigerants or coolant liquids, and in this case as well, a higher cooling effect and reduced operating costs can be expected compared to conventional air-cooled types.

[0029] In the liquid immersion cooling system according to the embodiment of the present invention, operational stability and equipment lifespan can be improved by immersing the equipment in liquid. This is achieved mainly by improving the following three points, which are factors that shorten operational stability and equipment lifespan: ・By immersion, the operating temperature of the equipment is stabilized throughout the entire equipment. The operating temperature of the entire equipment is always kept constant, and the operating temperature is maintained stably even when the equipment is under high load. ・By immersion, the number of moving parts (fans, etc.) in the equipment is minimized, so a reduction in malfunctions in moving parts can be expected. ・By immersion, the opportunity for contact with and accumulation of dust and dirt that is normally in the air is minimized, so short circuits caused by dust and dirt on the circuit board and unstable operation due to circuit board contamination are reduced.

[0030] Thus, according to the liquid immersion cooling system of the first embodiment of the present invention, the overall cost can be reduced by using a circulating configuration with natural water such as groundwater (river water, seawater, dam water, etc.), and the cooling efficiency can be improved, the safety of operation regardless of the load condition of the equipment can be improved, and the lifespan of the equipment can be extended.

[0031] Figure 4 shows and explains the configuration of a data center equipped with a liquid immersion cooling system according to one embodiment of the present invention.

[0032] As shown in the figure, the data center 100 is equipped with multiple immersion cooling systems 1-1, 1-2, ... 1-x, which can be centrally managed by the control device 20. The control device 20 is connected to the server device 30 via a communication network 40 such as the Internet. The control device 20 periodically or in real time transmits information such as the operating status of each immersion cooling system 1-1, 1-2, ... 1-x. The server device 30 receives this information such as the operating status and can remotely monitor the operating status.

[0033] <Examples> The results of the demonstration experiment of this system are described below.

[0034] This experiment was conducted by installing a liquid immersion cooling system in a server room within an operational data center. The data center is located near a nuclear power plant, providing a stable power supply and a relatively constant ambient temperature.

[0035] The main hardware and monitoring methods are as follows: The system is designed to accommodate up to eight 2U-sized GPU servers (e.g., NVIDIA H100) in a standard rack. However, for this evaluation, six ASIC machines (Antminer L9) for blockchain mining were connected to measure the cooling efficiency under continuous high load 24 hours a day, 365 days a year. This mining workload generates high heat throughout the year, allowing for a detailed understanding of the cooling performance and thermal characteristics.

[0036] More specifically, the details are as follows: A. Experimental Environment a1) Computing Hardware (ASIC Miner): Antminer L9 (Scrypt algorithm) x 6 Firmware: Antminer-L9-CV-release-202408211920.bmu Hash Rate: 16 GH / s Power Consumption: 3360W per unit (±5%)

[0037] a2) Immersion cooling system: Original single-phase system using ENEOS IX Type J oil Characteristics of the cooling liquid at 40°C: Viscosity: 0.025 Pa・s Thermal conductivity: 0.12 W / (m・K) Specific heat: 2.0 J / (g・K) Density: 0.85 g / cm ３ (at 25°C) Flash point: 220°C

[0038] a3) Immersion tank: Made of SUS304 (internal dimensions: 609 mm × 1039 mm × 902 mm) Oil volume: 502 liters

[0039] a4) Cooling water: Originally designed water-cooled heat exchange system using groundwater at approximately 14°C, flow rate 40 L / min

[0040] a5) Monitoring: The following items are measured through 24-hour continuous monitoring Power consumption: Power of ASIC and the entire system is measured with a power meter (3300 watt meter) Temperature: Temperature of immersion oil, ASIC surface temperature, inlet and outlet temperature of cooling water, etc. System stability: Hash rate, error rate, etc. are logged every 1 minute

[0041] B. Monitoring and Data Collection Through the above monitoring system, actual operational performance indicators were constantly measured and recorded.

[0042] C. Hardware Configuration ・Modular container unit (20ft container, internal dimensions 2.33 m × 5.867 m × 2.55 m) ・12 immersion cooling systems can be installed per unit (only one unit was used in the experiment) ・Maximum power capacity of 0.5MW (supplied at high voltage 6.6 kV) ・Heat exchange system using groundwater at 14°C

[0043] D. Purposes: Evaluation of thermal characteristics and energy efficiency Measure heat dissipation performance, temperature uniformity in the tank, thermal resistance of the coolant, PUE of the entire system, COP of the cooling system, etc.; Evaluation of operation stability of ASIC miners Monitor behaviors during continuous operation, including hash rate stability and error rate; Insights on implementation optimization obtained from actual operation Present precautions and improvement suggestions for design and operation. As described above, this experiment is valuable in that it measures and analyzes the effect of immersion cooling under actual operating conditions different from laboratory conditions, and provides specific implementation guidelines based on the measurement and analysis results.

[0044] E. Results To streamline the operation and management of data centers, a monitoring system was developed to track the data processing capacity, power consumption, oil temperature and other parameters of the immersion cooling system. Data processing capacity of the system: Six ASICs (16GH / s each) achieved stable operation at a total hash rate of 100GH / s. Each system consumes around 3300W per unit, and delivers performance as specified during 24-hour continuous operation.

[0045] As a major feature, unlike air cooling, this solution does not require large-scale air conditioning equipment, so additional power consumption other than that of the immersion cooling system itself can be suppressed. Since the experiment was conducted in winter (December, with an outdoor temperature of 2 to 10°C), there was almost no need to use additional cooling equipment. However, additional countermeasures may be required in midsummer.

[0046] It is particularly noteworthy that the system's PUE (Power Usage Effectiveness) reached an extremely low value of 1.03. Generally, the PUE of most air-cooled data centers ranges from 1.5 to 2.0, which confirms the significant energy-saving effect of the immersion cooling method. ・Thermal monitoring: The oil temperature in the tank is stable in the range of 33°C to 50°C (see FIG. 5(a)). Under high load, the maximum temperature is generally around 50°C, and it can be cooled to around 33°C through heat exchange with 14°C groundwater. This suggests that the cooling capacity can still meet the requirement with sufficient margin when 8 ASICs are operating at full load (see FIG. 5(b)). ・Power monitoring: Simultaneously record the power consumption of the ASICs and the entire system; ・Data logging: Automatically record data such as hash rate, power consumption and temperature every 1 minute

[0047] Here, Figure 5(a) shows the change in oil temperature within the immersion cooling system, and Figure 5(b) shows the data throughput of the immersion cooling system. As shown in these figures, the oil temperature was maintained within a sufficiently acceptable range even under full load for extended periods, confirming the continuous operation of the system.

[0048] A temperature sensor installed in the immersion tank provides real-time oil temperature data, and the system maintains a stable temperature of approximately 38°C. Typical commercial components operate within a temperature range of 0-70°C, and industrial components within -40-85°C, demonstrating that there is ample margin from both standards. Maintaining an oil temperature of 38°C ensures that the semiconductor surface temperature remains well within acceptable limits. This system utilizes groundwater at approximately 14°C for heat exchange.

[0049] A key feature of the single-phase cooling system is that it pumps circulating groundwater into heat exchange pipes, where it efficiently exchanges heat with the oil flowing through them. Since the groundwater, having lost its heat, is returned to the ground, this system can be circulated almost indefinitely in Japan's environment, which has abundant groundwater. This will lead to the realization of data centers utilizing renewable energy.

[0050] <Analysis> The four-hour service outage at OpenAI in early 2024 highlighted once again that "AI is becoming an infrastructure for society." Unlike traditional internet outages, when AI services stop, many operations and services are affected on a wide scale. This is due to the 1:N architecture (many users utilizing one large-scale model) and can be seen as a weakness of centralized AI infrastructure. This is where the decentralization of small-scale edge data centers emerges as a means of mitigating single-point failures.

[0051] The container-type liquid immersion cooling system of the present invention can be said to be a promising means of enabling its distributed deployment. It is particularly remarkable in terms of space efficiency, using a 20ft container (approximately 2.33 m × 5.867 m × 2.55 m, volume approximately 34.9 m³). ３ While this system can be configured in a small space, an equivalent air-cooled system would require a large server room with a ceiling height of 4.5 m x width of 8.5 m (at least 89.1 m²).３ In addition to the outdoor cooling tower space (180 m ３ The above may also be necessary.

[0052] While simple comparisons require assumptions, overall, a nearly five-fold reduction in space can be expected, resulting in reduced construction costs and shorter deployment times. Combined with a high energy efficiency of 1.03 PUE (significantly lower than the typical 1.5-2.0 for air-cooled systems) and improved thermal efficiency, this provides a significant advantage for edge data centers. Furthermore, the containerized design allows for construction in just a few months, significantly faster deployment compared to the 3-5 years required for conventional systems.

[0053] Finally, deploying a distributed AI infrastructure at the edge offers the following advantages: • Increased autonomy for local governments: By operating AI models tailored to regional characteristics, disaster response and public services can be strengthened. • Increased resilience at the national level: Dependence on centralized systems decreases, enabling the creation of an infrastructure that is more resistant to attacks and failures. • Improved corporate competitiveness: By placing data centers in the company's own environment, dependence on third parties is reduced, resulting in benefits in terms of both cost and security.

[0054] As reliance on AI and IoT increases, data centers are required to be even more efficient and sustainable. The liquid immersion cooling system according to the embodiment of the present invention demonstrates that container-type liquid immersion cooling technology is a powerful means of meeting these demands. Compared to air cooling methods, this system offers several advantages, including significant space savings, energy savings (reduced PUE), and reduced construction costs and time. Furthermore, by combining it with a distributed architecture, additional benefits such as fault tolerance and regional optimization can be enjoyed.

[0055] Although embodiments of the present invention have been described above, it goes without saying that the present invention is not limited thereto and various improvements and modifications are possible without departing from its spirit.

[0056] For example, it is possible to use refrigerants other than groundwater in conjunction with water from natural water sources.

[0057] 1...Liquid immersion cooling system, 2...Liquid immersion tank, 3...Heat exchanger, 4A...First water tank, 4B...Second water tank, 5...Natural water source, 6A, 6B...Water supply pump, 10...Rack, 11...Circulation unit, 12...Circulation unit, 13a...Inflow channel, 13b...Discharge channel, 14...Heat exchange system, 15...Circulation pump, 20...Control device, 30...Server equipment, 40...Communication network, 100...Data center.

Claims

1. An immersion cooling system comprising: an immersion tank containing equipment to be cooled and filled with a highly insulating liquid agent; a first water tank for storing water from a natural water source; a heat exchanger for performing heat exchange between the water in the water tank and the liquid agent; and a second water tank for storing the water used in the heat exchanger, wherein the first water tank discharges water supplied from the natural water source to the heat exchanger, and the second water tank receives the heated water from the heat exchanger, discharges a portion of it to the first water tank, and discharges the rest to the natural water source.

2. The liquid immersion cooling system according to claim 1, wherein the natural water source is at least one of groundwater, river water, seawater, or dam water.

3. The immersion cooling system according to claim 1, wherein, when a server device is housed in the immersion tank as the equipment, the fan mechanism is removed before housing the server device.

4. A data center equipped with the immersion cooling system described in any one of claims 1 to 3.

Images