CMS completes a new DAQ Server room in preparation for Hi-Lumi

By Attila Racz

The increase in data processing capabilities means that CMS is one step closer to being ready for Run 4.

The new Data Acquisition (DAQ) room is a state-of-the-art server room, essential for the Hi-Lumi phase of the detector and beyond. New server racks and free-cooling systems have been installed with these requirements in mind, allowing the finished DAQ server rooms combined to handle 50 Tb/s and host up to 5 megawatts of computers and servers by the start of Run 4.

During the Hi-Lumi phase of the LHC and the CMS detector, there will be an enormous increase in data requirements for the physics program. Today, the High-Level Trigger (HLT) selects which collision data are useful for analysis based on complex requirements, and does so at a rate of 2 terabits per second (Tb/s) for Run 3. This is already an enormous amount of data; a household using fibre optics for internet will have around 100 megabits per second available to them - more than enough for each member of a family to be online or streaming at the same time. What the current DAQ system handles is 20,000 times that capability, and this will only increase.

computing,servers,data,high luminosity,hi-lumi,hl-lhc — Scene from the inside of the new DAQ room (Image: S.Hurst/CERN)

Indeed, 2 Tb/s is only a fraction of what will be needed going into Run 4 at the beginning of the High-Luminosity phase: an incredible 50 Tb/s will be required. This huge increase in data is due to several factors, including the increase of collisions every time one proton bunch meets another. Currently, there are around 50-60 collisions every time two bunches meet; Hi-Lumi will have around 200!

On top of this, there will also be data coming in from more detector channels, and an increased rate of data from reading complete information from some sub-detectors, compared to compressed information. It is a bit like analysing images from the James Webb telescope compared to the lower-fidelity images from the Hubble telescope!

In order to be able to deal with this vast amount of data, this new DAQ room has been added to an already existing room, located one floor above. The latter will undergo extensive refurbishment between Run 3 and 4 as well, resulting in 89 high-power capacity water-cooled racks. The space used for the new room was the location of the legacy CMS control room, now transferred to a purpose-built new building nearby – you can read more about the new control room here.

CMS’ original DAQ room also hosts vital functions for the whole CMS site, including access controls, databases, user filers, network star point, safety-related servers, etc. This new room must replicate those functions before any refurbishment of the original can start, and CMS becomes ready for Hi-Lumi.

The new DAQ room is approximately 150 m2 in area, and contains 36 water-cooled server racks. The servers need to be kept cool in order to function, and the room will have an overall maximum cooling capacity of 2 MW, resulting in a power density of 13.5 kW/m2.

The water cooling production system itself will be upgraded, bringing it in line with the most efficient data centres around the globe. Currently, mechanical chillers are used to cool the servers. After refurbishment, the cooling water will be produced directly by water towers on the CMS site. The choice of high-performance racks and water towers enables ‘free cooling’: the outside air is used to cool the water before it is used to cool the servers. Given the climatic conditions of the Geneva area, the water temperature produced by the cooling towers is compatible with the operational conditions required by the servers even in the hottest months of the year. Significant energy savings will be seen throughout the year, and greenhouse gas emissions will also be far lower than conventional cooling methods.

Power Usage Effectiveness (PUE) is a metric used to quantify the energy efficiency of data centres. The theoretical best PUE is 1, meaning that all power is used by the IT equipment and no power is used by the infrastructure. The average PUE for Data centres around the world is typically around 2. Although the new DAQ room is already very efficient, with a PUE of 1.5, the move to free cooling will decrease this to a very good value of ~1.1-1.2!