The newly refurbished Tracker Cooling Plants

A series of important milestones have been passed during the last 3 months. With the delivery of refurbished cooling systems, pixels and strip systems have been brought back into operation after long shutdowns. Pixels has been operating since reinsertion of FPIX in April, and has been running at 4°C since May 16 when the bulkhead thermal screen was commissioned. More recently, on June 10 the Strip Tracker was powered up in its entirety, with cooling fluid circulating at 4°C, allowing commissioning of the Strip Tracker to proceed at full speed. The full Tracker is well on course to be ready for CRAFT, with Strip Tracker readout operation in ‘peak’ mode remaining also on track to be ready for beam operations in the Autumn in ‘deconvolution’ readout mode.

The main Tracker activity during the shutdown was the cooling plant refurbishment for Strips and Pixels systems. The objectives were to reduce the serious leaks observed in 2008 and improve the longevity and performance of the cooling system. This project began in December, involving the effort of around 10 engineers, several physicists and a large team of technicians. The project was very well managed from the beginning, with excellent quality control at each stage, delivering the working cooling plants in time for Pixels and Strips commissioning on a schedule that is compatible with being ready to provide tracks in CRAFT. The contributions of the CERN EN/CV-DC team were exemplary; the team managed the checkout and recommissioning of the cooling systems in a very efficient manner, being able to cover both the demands of CMS and ALICE, and meeting the tight schedule imposed by the Tracker.

Leak tests of the completed cooling plants after filling with C6F14 coolant showed no significant leakage on either Pixels plant or SS1 (silicon strip 1) plant, with SS2 leaking at only 250g/day. These figures mark a resounding success in attaining the primary objective of suppressing the leaks we experienced last year. The full Pixel system plus cooling plant leaks at a stable rate of ~60g/day. The measurements on Silicon Strip system exclude so far 6 cooling lines to the Strip Tracker, which are suspected or already known to leak away from the cooling plants. The actual situation is that with 179/180 lines operational for the silicon strips the total leak rate is ~5kg/day. This is considered to be sustainable for the current foreseen long run. In any case, measurements will continue in the background of detector commissioning to identify the precise leak rates in each of these remaining lines. One line will certainly be kept closed during operations up to the next opening of CMS as the leak rate is tens of kg/day. After CRAFT09, we aim to recover 45 single-sided modules of TIB layer 3 (representing 0.3% of the Strip Tracker) normally cooled by this line, by making use of the cooling from neighbouring parts of the system. Preliminary measurements of temperatures on the uncooled front-end hybrids were encouraging.

The forward pixel detectors FPIX were removed, maintained and reinserted according to the CMS planning during March and April. The majority of the 6% known faulty channels were recovered by replacement of the faulty cables and connections where shorts had developed. Another main action done was the replacement of the internal cooling pipes with stronger parts. In addition, the thermal contact to the readout optohybrids was improved by addition of ‘cold-fingers’. The whole removal, transport and reinsertion procedure was well organized and went very well, followed by rapid checkout of the connections made after re-insertion. All the details of the preparations and actions inside YB0 were noted by Technical Coordination in order to document the steps and procedures required for future maintenance scenarios, such as removal of pixels for the bake-out of the beam-pipe.

Another large work package during the maintenance period, associated with running the Tracker safely at lower temperatures, below the ambient cavern dew-point, was the sealing of all cold volumes at the detector (PP1 patch-panel boxes, cable channels, and Tracker bulkheads) and improvement of the dry air distribution. The objective of this work package was to be able to safely operate the Tracker at low temperatures regardless of whether CMS is open or closed, a situation that will be eventually essential for the longevity of the Tracker after there has been significant exposure to radiation. Extra humidity probes have been added in these volumes and repairs were also made to the accessible faulty connections in the thermal screen, which is also being commissioned, along with the cable channel heater wire system. These two environmental systems are not essential for running at 4°C but the ALARA principle, and more importantly our motivation to complete the entire Tracker system, means we aim to commission also these subsystems as soon as we can.

The nominally cold and dry volumes of the Tracker system have been demonstrated to be effectively sealed and flushed against ingress of external humidity. The dewpoints are at acceptable levels for running at 4°C during 2009/2010. After the switch on June 10 to flushing YB0 with nitrogen, following the closure of the central detectors when the first endcap disks were locked, the dewpoints within the Strips and Pixel internal volumes fell to below -40°C, lower than -15°C at the Tracker bulkheads and lower than 5°C at the PP1s and cable channels. The Tracker bulkheads will need some further work to be compatible with future operations having coolant circulating at -20°C; work which can be postponed comfortably up to the next shutdown, when we will also wish to test the maximum stable heat removal capacity of the Tracker cooling system.

All the activity that took place on the Tracker system parts inside YB0 was finished to a high standard, by a small team of people experienced in working with care inside YB0 around the beam-pipe. It should be noted also that these maintenance tasks were done completely in the shadow of other work driving the schedule at P5, in particular ES installation, Pixel removal and reinstallation, and the opening and closing of the barrel wheels and endcaps. This achievement was possible only due to the excellent attitude and flexibility of the staff involved and we also appreciated very much the support of the P5 technical staff in providing good access to the Tracker bulkheads and PP1s.

The integration of Strips and Pixels DCS has been completed and deployed in time for the Strip Tracker commissioning. Pixels commissioning started earlier with the older pixels-only package but has recently adopted the Tracker DCS package. The readiness in time for Strips commissioning and the smoothness of the transition for pixels operation is a credit to the staff involved and the entire Tracker community and CMS will profit from this integration work. The team of DCS experts for Strips and Pixels has now been consolidated around one sophisticated, easy to use, well supported system, that now has improved maintainability in the longer term. The majority system developed by the Tracker has been adopted by CMS DCS, which will prevent small problems, such as single trips, from putting a full TTC partition into an error state.

A large effort, detailed in the following paragraphs, has been made in recent months to improve the Strip Tracker online and commissioning software, specifically in the areas of the online diagnostic system, FEC operation, online databases, O2O, TrackerWBM, maintenance of the CMSSW commissioning code, development of the TKCommissioner tool, and data file archival. All the developments and critical items were completed and validated prior to Tracker commissioning using test systems in Meyrin, and then the first half of the Strip Tracker that was available after the first cooling plant was recommissioned.

The Strip Tracker online diagnostic system has been fully deployed, managing correctly the error messages and archiving them in a dedicated database, for which an interface is available and a full browser is under development. FEC operation has been improved, with implementation of redundancy, smart configuration of modules and ability now to push all DCU values to PVSS. Online databases improvements include module masking, which has been used successfully in commissioning activities to isolate problematic modules, and functionality for error reporting system and problem tagging via a ticket system. Within O2O activities, masking of Strips modules has been implemented and history of O2O transfers is in place. The TrackerWBM package has been deployed in the secured P5 network, and it has started to be used as a tool for commissioning, with the development of many useful plots.

The CMSSW commissioning software was already in a good state last year, and has since been well managed and maintained with minor developments made to integrate improvements in noise measurements and bad channel masking. A powerful tool, the tkCommissioner GUI, has been developed and is now in use. It launches commissioning analyses, uploads results to the databases, allows powerful examination of the results and enables tagging of problems through a user friendly interface. Scripts have been developed for archiving of commissioning data, accounting of the files and cleanup of disks.

The time taken to configure and initialize the Strip Tracker has been measured, and means of optimizing it were implemented. Configuration time in 2008 was around 240 seconds for the whole Tracker, whilst other sub-detectors took less than 100 seconds. Early tests indicated that the new improvements are able to bring the time down to about 120 seconds. In contrast, initialization time for the Strips is only ~10 seconds whilst up to 70 seconds for other subdetectors, therefore it is being considered how to balance better these two steps for the Strips.

Software is also largely ready for the FED spy channel, which is the parallel readout path of event raw data frames via VME. The spy-channel software architecture allows monitoring of pedestals, noise and gains during global runs and beam data taking conditions. From the CMSSW analysis point of view the treatment of spy channel data will be identical to analysis of similar data taken over the standard readout. There is work still ongoing on the packaging of the spy channel data and the effort required to make the firmware modifications needed to trigger the spy readout path in a synchronous manner across all FEDs, which is an attractive proposition, is being evaluated.

Strip Tracker firmware is still undergoing revision, including the strip Tracker FEDs readout and APVE (APV front-end emulator) in order to remedy several problems experienced in 2008. More work is needed to arrive at ‘zero suppressed light’ readout of Strip Tracker FEDs at 100kHz. For the APVE, a veto has been introduced to exclude a small fraction of noisy events. This feature is now under test in Meyrin on a small scale Tracker test system before being tested and deployed at P5.

Another online commissioning tool was quickly developed to allow the ‘warm’ checkout of the Strip Tracker connections during May, before YB0 was closed, when we were still without cooling. The procedure rapidly powered-up, scanned and then powered-off Strips front-end modules one power-supply unit at a time (~10 modules), providing full information of which modules responded and measuring voltage in realistic conditions (with front-ends active), whilst avoiding any significant increase in temperature at the front-end hybrids. The checkout showed that 15128 out of 15148 Strips modules were able to communicate, that all Strip Tracker power and control cabling remained well connected, confirming that the work to improve sealing and dry-air distribution inside PP1 patch panels was indeed done very carefully without disturbing any electrical connections. In addition, we were able to verify the entire map of connections between control system, power supplies and Strips frontend modules, identifying clearly a small number of swapped cables, which have now been fixed or properly taken into account to allow unambiguous operation of the affected parts of the system.

The recently started commissioning activity of the Strip Tracker is progressing very well and is on schedule during the first two weeks. Tracker partitions will be included into coming global runs prior to CRUZET and CRAFT to exercise the full chain of the Tracker readout. It remains to be seen how much tuning will still be required at the start of CRAFT in order to bring the strip Tracker into readiness to operate in ‘deconvolution mode’, where output signals from the APV frontend chips are confined to a single bunch crossing. This is the intended mode of readout for beam operation in 2009. It is planned to make the necessary fine timing adjustments layer by layer during cosmic running. This will not affect the overall tracking performance for what concerns CMS global needs and will put the Strip Tracker in a position to make the switch to deconvolution mode during CRAFT and profit from substantial data taking periods in both peak and deconvolution modes.

The Tracker maintenance and operations activities have undergone a recent series of reviews to assess readiness for 2009 operations and to acknowledge formally the work done, identify action items and prioritize the remaining work. The review agendas are posted in indico at
http://indico.cern.ch/categoryDisplay.py?categId=872
The members of the review panels from outside the Tracker community are warmly thanked for their participation.

Submitted by K. Gill, June 2009