Report Summary

This document provides design recommendations for the new *** Data Center focusing on the following three (3) current project timeline constraints listed in the Data Center Design Review Statement of Work submitted to ****** (Addendum 1).

• Rack Positioning
• HVAC Airflow
• UPS Power

These three (3) items were identified as design priorities that affect all other items listed in the SOW.

Rack and Cabinet Positioning

The terms rack and Cabinet are sometimes used interchangeably, which is incorrect. Computer cabinets are fitted with doors and side panels, which may or may not be removed, and are available in a very wide variety of sizes and colors. Most cabinets provide connections for electrical power. Some cabinets provide fans and baffles designed to move cooling air in a specified direction and often, at a specified rate. Most cabinets draw in ambient air for cooling from the front and discharge heated exhaust air to the rear. Cabinet doors should at least be 63% open with a perforated pattern and maintain a minimum 1.5-inch) clearance between the systems and any front or back cabinet doors to allow adequate airflow.

Cabinets enclose a rack, which is a frame that provides a means for mounting electronic equipment. Racks can also stand-alone and do not require the doors, panels and other integrated equipment that comes with cabinets. Racks come in different types. One type consists of four (4) vertical rails, which may or may not be enclosed by cabinet doors and panels. Another, and more common type, consists of two (2) vertical rails, which are not enclosed by cabinet doors and panels. These are used mainly for telco equipment.

All types of racks and cabinets are currently used in the active old data center at ***. There is no standard size or manufacture set forth. With this in mind it is difficult to layout the footprint for the equipment in the new data center. Additionally, the low ceiling and exposed overhead floor beams create thermal pockets that can and will impede the hot return airflow to the CRAC (computer room air conditioners).

A typical cabinet occupies 12 square feet of floor space, which corresponds to three tiles, each tile measuring 2x2 feet. When room for aisles, power distribution equipment, air conditioners, and other facility equipment is included, floor space utilization may equal 20 square feet, or five tiles per cabinet.

When possible, rows of racks or cabinets need to be perpendicular to the CRAC units. This formation facilitates airflow down the aisles to the CRAC return ducts. Heated air should have an unobstructed path back to the CRAC units return ducts .. Heated air must not be forced to travel over cabinet's rows to get to the CRAC return ducts. Doing so could heat the air in the cool aisles

Because of the front-to-back airflow of the systems, the ideal placement of the cabinets and racks have the systems installed front to front and back to back. This configuration eliminates direct transfer of hot exhaust from one system into the intake of another system. Locate air distribution tiles so that conditional air can be delivered effectively to the intake of each cabinet. (ref . To drawing 1 and drawing 3)

In order to allow for installation, removal or maintenance of a server or other equipment, a clear service area must be maintained in front and back of the cabinet or rack. At a minimum, this area should extend 3 feet forward from the front of the cabinet or rack and 3 feet on either side of the server when it is fully extended from the rack. You should also keep at least a 3-foot clearance at the rear of the cabinet or rack to allow for service and maintenance. ( ref . Fig 1)

There are no side clearance requirements for cabinets or racks because the air intake for the servers is from the front and the exhaust is to the rear. Where possible side panel should be removed to allow cable runs between cabinets. Because of the various sizes and exhaust airflow pattern of the cabinets in use, this may not be feasible. Additionally, all high exhaust temp cabinets should be placed as close as possible to the CRAC units with the taller cabinets further away from the CRAC units so that the airflow will have an increasing area to flow into the intakes of the CRAC units. Additionally shorter cabinets should be placed in the cabinet row under the overhead floor support beams to provide as much area for airflow as possible. ( ref . To drawing 2)

Because of the restricted airflow between the overhead floor beams, a thermal layer will exist that will cause a thermal transfer to the second floor. This may cause a problem maintaining proper temperature control in this space. Insulation will help control this problem but it should not extend more than 3 inches below the second floor deck and the beams should be wrapped no wider than the top and bottom flanges. As much air space as possible needs to be maintained above the racks and cabinets. If a cabinet is equipped with a top exhaust fan, this should be placed under the space between beams where possible to help with maintaining airflow back to the CRAC units.

HVAC Airflow

Computer systems reliability is dependent upon a stable environment. The design of the environmental control systems for your data center must ensure that each system can operate reliably while remaining within the range of its operating specifications.

Accurate and comprehensive monitoring of environmental support equipment and in-room conditions is extremely important in a sensitive data center environment. The monitoring system should have historical trend capabilities. Analyzing historical trend information is instrumental when determining seasonal changes or other contributing influences as outlined in this document. Also, the environmental control system should have critical alarm capabilities. The system must be able to notify the appropriate personnel when conditions move outside of the systems' established operating specifications. While Servers can operate in diverse locations and within a wide range of environmental conditions, stringent control over temperature, humidity, and airflow is necessary for optimal system performance and reliability.

An ambient temperature range of 70 to 74°F is optimal for system reliability and operator comfort. While most computers can operate within a rather broad range, a temperature level near 72°F is desirable because it is easier to maintain a safe associated relative humidity level at this temperature. Ambient relative humidity levels between 45% and 50% are most suitable for safe server operations. This optimal range also provides the greatest operating time buffer in the event of an environmental control system failure. Further, this recommended temperature will provide an operational buffer in case the environmental systems are down.

Note that the operating temperature range for the server is either 41 to 104°F or 41 to 95°F. These temperatures apply to the air taken in by each server at the point where the air enters the server, and not necessarily the temperature of the air in the aisles. To check the temperature of the air entering the servers, measure the temperature at 2 inches from the front of the equipment. It is also important to verify that the temperatures midway, vertically and across the aisle are within the servers' operating temperature ranges. These measurements are necessary because temperatures in the data center are different depending on where in the room the measurements are taken and due to the unique restrictions in the airflow patterns within the data center. The heat load in the data center can vary depending on the density of heat-producing equipment located within the room. Aisle temperatures can give you a first-level alert to the conditions in the data center. Also avoid cooling short cycles, which can occur if the perforated tiles or grilled tiles are placed between the CRAC and the nearest heat-producing equipment. If tiles are laid out in that way, cold air returns to the CRAC without circulating through the equipment. The CRAC might register that temperatures in the room are cooler than is actually the case. The CRAC might cycle out of its cooling mode while temperatures in the room still call for the cooler air. CRAC units must be placed perpendicular to the cabinet rows so that the heated return airflow does not have to cross over another row of cabinets. Ensure that the CRAC equipment can adequately move air down the aisles so that heated air does not flow over the cabinets and racks to the front of the systems (ref. Drawing 1)

The high-density heat load in the data center can require as many as 30 air changes per hour. If airflow pressure is inadequate, the conditioned air will heat up before it reaches the area in need of cooling. Measure airflow speed in different zones of the floor to determine whether the existing airflow pressure is sufficient to provide the necessary conditioned air to the systems. Take measurements every 13 to 16 feet. Measurements taken at lesser distances might not be able to detect a significant pressure difference. The recommended airflow speed range is between 10 to 13 feet per second.

The Major constraint for conditioned air is the lack of headroom in the data center. The total raw height is 10 feet of which existing second floor support beams and piping reduces that to 8ft 9in. In addition a raised floor 8in high reduces the free space to 8ft 1in. Current practice calls for typical raised floor heights of 12-18in. A recent full-scale field experiment has found that low-height under floor plenums (7in. and lower) can provide uniform airflow performance across a 3,200-ft2 area of a building. The report recommends that, on average, at least 3in. of clear space for airflow should be provided in addition to the height required for other factors (conduits, piping, cabling, etc.). During installation of the raised floor ensure that they caulk all floor to wall joints, penetrations and where tiles meet perimeter equipment to minimize air loss. All electrical connection should use low profile flush mounted boxes of the style shown in fig. 2.

This will help ensure that the conditioned air will end up where it is needed. Any opening in the floor tiles should be of a type that restricts air leakage. Addendum 2 contains the product specification guide from one supplier for floor access boxes, grommets and air diffusers. The small air diffusers can be used around and under cabinets that have a high heat load where a full air diffuser floor tile will be too large or may cause return airflow problems.

The current design of the data center calls for 4 15-ton CRAC units. Two will penetrate thru the west wall and two Liebert “in room” units will be moved from the old data center. This is a total of 60 tons rating. The archaic terms of BTU and Tons will be phased out over time. (The term “Tons” refers to the cooling capacity of ice and is a relic of the period from 1870-1930 when refrigeration and air conditioning capacity were provided by the daily delivery of ice blocks.) For this reason, this document will discuss cooling and power capacities in Watts . For this reason the following conversions are provided:

Given a value in	Multiply by	To Get
BTU per Hour	0.293	Watts
Watts	3.41	BTU per Hour
Tons	3,530	Watts
Watts	0.000283	Tons

60 tons rating converts to 211,800 watts of cooling. Rounded down rule of thumb to 211k watts. Total square footage of the data center is approx. 1,983-squared ft. at this time. Removing the ramp space and other dead space rounded down rule of thumb to 1,800-squared ft. That comes out to 117 watts per square ft of cooling. Typically, a cabinet footprint requires 12 square feet. However, cooling measurements are calculated using the gross squared footage required by the cabinets or racks, which is not just the area where cabinets or racks are located. The measurement includes aisles and area where power distribution, ventilation and other facility equipment are located. Gross square footage is estimated to be 20 square feet per cabinet or rack. Based on 117 watts per square foot and 20 square feet per cabinet, each cabinet is allowed a cooling capacity of 2340 watts. Remember, 2 kW per cabinet in a data center is only an example. Some cabinets may and will require 3 kW or more of cooling capacity. With that in mind, it will be necessary to have several rolling cool units on hand during the move to the new data center to handle the temporary cooling load until the Libert units are moved and all equipment is operating normally.

The total heat output of a system is the sum of the heat outputs of the components. The complete system includes the IT equipment, plus other items such as UPS, Power distribution, CRAC units, lighting and people. The heat output of UPS and power distribution system should be removed from the total watts need for this data center as long as the plan to isolate the UPS system in its own space moves forward. Also for safety reasons this equipment should be in it own controlled and cooled space. ( ref . Drawing 1) In Addendum 2 are specs for modular walls that can be used to create a separate room for the UPS. This system can be changed and moved as needed to move or change out the UPS.

A detailed thermal analysis using thermal output data for every item in the data center is possible, but a quick estimate using simple rules give results that are within typical margin of error of the more complicated analysis. The quick estimate also has the advantage that it can be performed by anyone without specialized knowledge or training.

Item	Data Required	Heat output calculation	Heat output subtotal
IT equipment	Total IT power in Watts	Same as total IT load power in watts	67k Watts
UPS with Battery	Power system rated power in Watts	(0.04x Power system rating) + (0.06 x Total IT load Power	N/A Watts
Power Distribution	Power system rated power in Watts	(0.02 x Power system rating) + (0.02 x Total IT load power)	N/A Watts
Lighting	Floor area in sq ft	2.0 x floor area (1800 sq ft)	3600 Watts
People	Max # of Personnel in Data Center	100 x Max # of personnel 20	2k Watts
Total	Subtotal from Above	Sum of heat output subtotal	72,600 Watts

This is based on the current IT load as reported by the UPS system in the old data center. This amount will increase as new systems are put online and this figure is only an estimate. The true IT loads can only be known by a system audit.

This analysis ignores sources of environmental heat such as sunlight through windows and heat conducted in from outside walls. The data center is located within the confines of an air-conditioned facility; the other heat sources have been ignored.

The 2 new CRAC units in the new data center are rated at 105.9kW. With this info we can see that the load within the new data center with all the IT equipment operational is approx 68% of capacity of these two units. With the traffic going in and out and the lost of condition air through open doors and heat from personnel they will quickly reach max and may need supplemental units until the Liberts are moved in and operational.

Based on these figures, the new data center with all four CRAC units operational will have a total of 211kW of cooling with a IT load of 72.6kW of existing equipment, a operational 34% of available cooling load.

In addition to removing heat, a CRAC is designed to control humidity. Ideally, when the desired humidity is attained, the system would operate with a constant amount of water in the air and there would be no need for ongoing humidification. Unfortunately, in most CRAC systems the air-cooling function of the CRAC system causes significant condensation of water vapor and consequent humidity loss. Therefore, supplemental humidification is required to maintain the desired humidity level. Supplemental humidification creates an additional heat load on the CRAC unit, effectively decreasing the cooling capacity of the unit and consequently requiring over sizing. The required over sizing for a CRAC unit therefore may be up to 30%. Water lines under the floor to drain and feed the CRAC units are required. A water leak dedication system is required and needs to be installed. Water drainage will be needed and should be able to remove the water without it backing up and entering back into the data center.

UPS Power

The design of the electrical power system must ensure that adequate, high-quality power is provided to each sever and all peripherals at all times. Power systems failures can result in system shutdown and possible loss of data. Further, computer equipment that is subject to repeated power interruptions or fluctuations experiences a higher component failure rate than equipment that has a stable power source.

It is important to secure multiple sources of power when possible. Ideally, Multiple utility feeds should be provided from different substations or power grids. This setup provides power redundancy and backup. The new data center has only the one feed from the power grid. Therefore it is very important that the UPS system be capable of maintaining the critical load of the data center for a minimum of 15 minutes during a power failure. More time is better than less. The backup power generator should be able to carry the load of both the computer equipment and the supporting CRAC equipment. The IT load on the Model 80 UPS in the old data center is beyond the rated amount from Powerware. Design plans call for a new Model 160 to be installed in the new data center and the old unit moved to the new data center. The model 160 is rated at 144kW. The IT load at this time is approx. 67kW, 46% of the model 160. The plan to move the old model 80 over and supply power will not provide a redundancy system to maintain system uptime. Only a small percent of the systems will be keep online should the main Model 160 fail. An “N+1” redundancy system is required to maintain system uptime. An N+1 redundant power configuration does not add to the power capacity of the systems. “N” represents the number of UPS need to power a fully loaded system. The “1” means that there is one additional UPS to handle the load if a supply fails. In a 1+1 configuration (that is, two UPS are installed, Each capable of providing enough power for the entire system) both are turned and are delivering power. Each supply delivers 50% of the power need by the system. If one fails, the supply that is still online will deliver 100% of the power needed to keep the system running. With the current plan to install the new model 160 and move the old model 80, you cannot do either. An additional model 160 should be installed and the old model 80 can be sold. The parts and support for this model will grow less each year and many customers that are using the model 80 are looking for units to configure for an “N+1” or “1+1”. This option will give the data center enough capacity to grow for the short term. In drawing 1 I show two models 160 installed on the data center floor. They are in a separate room that is supplied conditioned air from a separate split system AC unit. UPS control units and battery cabinets should be in their own controlled space. The amount of heat added to the load in the data center with the low overhead and return airflow problems would not help with controlling the cooling load. Using the modular panels types as outline in addendum 2 would isolate the UPS' from the rest of the data center and allow the walls to be removed to access the units for major service. The class panels will also enable the staff to view the units without having to enter the room in case of a problem. A separate fire system or zone can be used. If a problem accrues, the whole data center does not need to be flooded with FM200. Separate PDU' (Power Distribution Units) two for each UPS should be installed on the data center floor. ( ref . Drawing 1) This will provide enough circuits for the IT equipment. Each PDU should be installed in the center of each cabinet row. With this layout the length of each power run can be keep as short as possible. Armored cable should be used under the floor tiles and each cable should be secured to the concrete floor to within 12 feet of the service outlet. Using the type of flush mount access boxes in addendum 2 that will keep the service outlet off the concrete floor and out of reach of any water leaks. With 12 feet of service loop the floor tile can be positioned easily if any changes to the data center layout is needed. To maintain a safe facility, the AC current draw should not exceed the maximum current limit for your power outlets. In the USA the maximum is 80% of the outlet's total capacity.

Conclusion

With the limited amount of time allotted for this design review these three items are the ones identified as constraints to the timeline. Not all situations were address and additional issues are outstanding. If we can be of any additional help let us know.

Stewart Irwin