Around the one-year anniversary of the first COVID-19 lockdown in Switzerland, physicists and engineers working on the Compact Muon Solenoid (CMS) experiment closed the case on an elusive problem affecting the operation of the Hadron Calorimeter (HCAL) detector.

The mystery began during data collection in 2018, when CMS lost communication with a key module installed underground in the HCAL Endcap detector.  Known as a next-generation Clock and Control Module (ngCCM), the unit is one of 36 identical devices that orchestrate data transmission for the HCAL Endcap on-detector electronics. The affected module was removed and tested in 2019, but no cause for failure was found. Later that year, communication to another ngCCM was inexplicably lost. The HCAL team noticed that both instances of communication loss were preceded by a strange symptom: an anomalous electrical current. After careful scrutiny, hints of this anomalous behaviour were observed in roughly half of all HCAL Endcap sectors. The pervasiveness of the symptom sparked concern, but the link between the anomalous current and the communication failures remained elusive.

In Spring 2020, following the CERN shutdown to “safe mode” due to the COVID-19 pandemic, the ngCCM investigation was among the first projects to resume as CERN gradually reopened. A small five-person crew continued making measurements at the CERN Prévessin site. As the crew followed strict safety precautions to prevent the spread of COVID-19, this task became one of the first pilot projects to explore how to accomplish work safely at CERN during the pandemic.

^{Team working in the HCAL ngCCM Laboratory. Sebastian (Xuli) Yan, Left. Grace Cummings, Right. Credit: S.J. Hertzog, CERN}

By the end of summer, the anomalous currents were traced to an optical transceiver located within the ngCCMs. Communication with the HCAL Endcap is done through optical fibers, and the optical transceiver converts light carried along the fibers to electrical signals. Having pinpointed the relevant component, the HCAL team sought to find a solution. A hint emerged within the design of the HCAL upgrade. Even though the HCAL Endcaps share their technology with the newly upgraded HCAL Barrel, completed in Fall 2019, the anomalous behaviour was largely absent in the Barrel. A crucial bit of evidence was secured during the first continuous two-week period of operation after the COVID-19 lockdown. Only one of 72 ngCCMs in the Barrel exhibited the anomalous current, and upon extraction, it was clear that something was unusual; it was accidentally missing a piece of copper designed to regulate the temperature of the optical transceivers in the Barrel ngCCMs. This discovery revealed the impact of temperature on the optical transceivers and suggested that a solution would involve the adaptation of the copper plates for the Endcap system.

^{Open HCAL Endcap next-generation Clock, Control, and Monitoring unit (HEngCCM). Built of four printed circuit boards (PCBs), the HE ngCCMs feature two clock (outermost) and two control (innermost) boards. Credit: S.J. Hertzog, CERN}

In December 2020 with this possible solution in hand, the HCAL team began the removal of the HCAL Endcap ngCCMs. Over the winter holidays, a dedicated team added features to improve heat flow and recertified the electronics to return underground. The ngCCMs were reinstalled in the complex 360° geometry of the HCAL Endap by HCAL team experts. The work was completed in March 2021. The first comprehensive tests revealed no sign of anomalous currents.

^{HCAL team members Maria Toms (left) and Irina Tlisova (right) beginning to remove the HCAL Endcap ngCCMs. Credit: J.M. Ordan, CERN}

Now, with the retrofit of the HCAL detector completed, members of the HCAL team continue to collaborate with other groups at CERN to explore the operation and performance of the optical transceivers. As this technology is used within the wider High Energy Physics community, the CMS HCAL experience is continuing to provide useful guidance to other teams of experimentalists.

^{Our rework team group photo, taken outside of the CMS HCAL Clean Room in CERN Building 904. Back Row, Left to Right: Long Wang, Sebastian (Xuli) Yan, and Furkan Dolek. Middle: Jay Dittmann. Front, Left to Right: Grace Cummings, and Fatma Boran. Not pictured, Markus Seidel and Irina Tlisova. Credit: S.J. Hertzog, CERN}