CFD investigations of data centers’ thermal performance for different configurations of CRACs units and aisles separation

==Abstract==

The thermal performance of data centers is numerically studied for different configurations of computer room air conditioning (CRAC) units and physical separations of cold and hot aisles. Temperature distribution, air flow characteristics and thermal management of data centers racks array are predicted and evaluated for the different arrangements. Measureable performance indices: supply/return heat index (SHI/RHI), return temperature index (RTI) and return cooling index (RCI) are used to measure the thermal management effectiveness of data center racks. The results showed that: (i) hot air recirculation, cold air bypass and the measurable performance indices of the racks strongly depend on the racks location in the racks array, (ii) the CRACs units layout affects the thermal managements of the racks array especially the sides and middle racks in the array, and (iii) using cold aisle containments enhances the thermal performance of the data center.

==Keywords==

Data centers’ thermal performance; Numerical study; CRACs arrangements; Cold aisle containments

==Nomenclature==

''Cp''- specific heat (J kg<sup>−1</sup> k<sup>−1</sup>)

CRAC- computer room air conditioning

''F''- body force (N)

<math display="inline">\overset{\cdot}{m}</math>- mass flow rate (kg s<sup>−1</sup>)

''T''- temperature (°C)

''T<sub>ref</sub>''- reference temperature (°C)

''Q''- power dissipation (W)

''U''- velocity (m s<sup>−1</sup>)

RHI- return heat index

RCI- return cooling index

RTI- return temperature index

SHI- supply heat index

''σ''- opening ratio of perforation tile

===Superscripts===

''r''- rack

''c''- CRAC

===Subscripts===

in- inlet

out- outlet

''ref''- CRAC supply

max-rec- maximum-recommended

max-all- maximum-allowable

min-rec- minimum-recommended

min-all- minimum-allowable

==1. Introduction==

The requirements of reducing IT server’s sizes, increasing server’s power density and maintaining energy efficiency represent a challenge of the cooling system design of data center. The cooling system must be designed to maintain the server’s operating temperature within the reliable limits whatever the server’s power density. At the same time the energy consumptions of the cooling system must be maintained as minimum as possible. Increasing server’s power density dramatically rises the energy consumption in data center cooling [[#b0005|[1]]], [[#b0010|[2]]] and [[#b0015|[3]]]. Efficient air distribution, cooling and thermal management systems in data centers are required to satisfy these requirements. Managing airflow for cooling IT servers enhances cooling system effectiveness and reduces energy consumption. Server’s racks and cooling units (CRACs) arrangements in data centers strongly affect the air distribution and thermal management of data centers. Hot air recirculation and cold air bypass around the server’s racks are the main problems of reliable operation and energy consumption of data centers. Hot air re-circulation from the rack rear to the rack front is expected in open aisle data centers ([[#f0005|Fig. 1]]), and causes non-uniformity and elevation of air temperatures at the server’s intakes [[#b0015|[3]]]. Some servers will receive cold air (bottom servers) and other servers may receive hot re-circulated air (edge and top servers) resulting in the server’s inlet air temperature exceeding the recommended limit (27 °C). Typically, pushing more supply air or lowering the supply air temperature is required just to keep these edge and top servers below this limit. In this case, the bottom servers will be over-cooled resulting in wasted cooling energy. Therefore, it is necessary to predict the non-uniformity in the air temperature distribution entering the rack and correct for this issue to ensure reliable operation of server’s racks.

<span id='f0005'></span>

{| style="text-align: center; border: 1px solid #BBB; margin: 1em auto; max-width: 100%;" 
|-
|


[[Image:draft_Content_229468638-1-s2.0-S1110016816300047-gr1.jpg|center|489px|Example of a typical data center [3].]]


|-
| <span style="text-align: center; font-size: 75%;">

Figure 1.

Example of a typical data center [[#b0015|[3]]].

</span>
|}

Most of the recent data centers studies are devoted to enhance the design to solve the air recirculation and bypass problems [[#b0015|[3]]]. There is a recent interest in predicting air flow and temperature distributions in data centers using CFD codes. Most of the CFD studies were validated using results of experimental studies of air temperature measurements in data centers. An early study on floor tile airflow was performed by Kang et al. [[#b0020|[4]]]. The study showed that a simple model of the raised floor plenum can predict the air flow distribution through the various tiles. Schmidt et al. [[#b0025|[5]]] studied the effect of plenum depth and perforation area on airflow distribution through the perforated tiles with different arrangement of data center using measurements and predictions via a compact model [[#b0030|[6]]]. Patankar [[#b0035|[7]]] carried out a CFD study for airflow distribution through the perforated tiles of raised-floor plenum. The pressure above the raised floor is assumed uniform to limit the domain of the study for the raised floor plenum. Abdelmaksoud [[#b0040|[8]]] showed that the strongest factors influencing and improving the data centers CFD simulation results are the inclusion of correct tile flow model, buoyancy, and realistic turbulent boundary conditions. A practical tile flow model using momentum source was developed to correct the global values of mass and momentum of air jets issuing from the perforated tiles. Kim [[#b0045|[9]]] conducted a study on the distribution of cooling air flow in data centers for two of the most promising types of the floor tiles; fan-assisted tiles and tiles with louvers. The fan-assisted tile was sits under a traditional perforated tile in order to enable a variable local tile flow rate. The study recommended that the active tile can be used with a control system to avoid hot spots by varying the local flow rate at a particular rack location depend on a monitored temperature or some other variable. Schmidt and Cruz [[#b0050|[10]]] studied the effect of air flow distribution from the perforated tiles on the racks air intake temperatures using CFD tool called Tileflow. The effects of the plenum and opening ratio of the perforated tiles rack intake temperatures were investigated. Cho et al. [[#b0055|[11]]] numerically studied the performance of the cooling system and IT racks thermal management in a data center of high power density. Cooling efficiency of different configurations of air distribution systems are studied and compared using CFD analysis. Schmidt [[#b0060|[12]]] used CFD modeling to numerically study the effect of the different design parameters in a real data center on air flow uniformity from the perforated tiles. The CFD results were validated against experimental tests results. Bhopte et al. [[#b0065|[13]]] studied the effect of plenum height, ceiling height and location of floor tiles on data center air flow distribution and rack inlet air temperature for 12 kW racks power density. Patankar [[#b0070|[14]]] and [[#b0075|[15]]] studied the effects of the opening area of the perforated tiles and plenum depth on airflow distribution using an idealized one-dimensional CFD model and other various techniques. Sharma [[#b0080|[16]]] studied the effects of the widths of cold and hot aisle and ceiling height on the thermal performance of data centers. The study showed that optimization of data centers cooling can be achieved using the dimensional parameters supply heat index (SHI) and return heat index (RHI). Zhang et al. [[#b0085|[17]]] studied air flows characteristics through the perforated tiles of a plenum test facility. Ibrahim [[#b0090|[18]]] studied the effects of power density variation, airflow rate mass of IT equipment on data center thermal performance. The results showed that the IT equipment mass plays an important role in how slowly or quickly the facility temperature falls or rises. More recently, Nada et al. [[#b0095|[19]]], [[#b0100|[20]]], [[#b0105|[21]]], [[#b0110|[22]]] and [[#b0115|[23]]] studied the effects of perforated tiles opening ratio, racks power density, racks power distributions schemes and using cold aisles partitions and containments using a scale physical and CFD models for data center.

The above literature illustrates that many researches have included the under floor plenum in the CFD simulation in order to investigate the perforated tile air flow distribution assuming a uniform pressure in the data center room as its variations are very small compared with perforated tiles the pressure drop. Other researchers have modeled the plenum only without simulating above the floor and studied the effect of perforation area and plenum height on air flow distribution. Detailed comprehensive study on the effects of server’s racks and CRACs distribution and arrangements on the air distribution, temperature distribution and thermal management in data centers are not available. In the present paper, the effects of data centers racks and CRACs distribution and arrangement and the effect of using different configurations of cold aisle containment in each arrangements on the air flow characteristics and thermal performance of the data centers are to be investigated. Supply/return heat indices (SHI/RHI), return temperature indices (RTI) and return cooling indices (RCI) are used as measureable overall thermal performance parameters of data centers.

==2. Physical model==

[[#f0010|Fig. 2]] illustrates the schematic of the physical model considered in this study. A data center room of dimensions 6.71 × 5.49 × 3.0 m was considered as the physical model of the present investigation. The data center houses 14 server’s racks, each has a power of 3.5 kW. The racks are arranged in two rows, each row houses seven racks, with a spacing 1.22 m between the two rows. The racks rows are arranged to be at 1.22 m distance from the room walls. A typical rack dimension (''W'' × ''D'' × ''H'') of 0.610 × 0.915 × 2.0 m is considered in this study. Each rack is assumed to include four server’s chassis of dimensions 0.610 × 0.915 × 0.5 m. To model the server’s heat generation, heat generation was considered at the server surface.

<span id='f0010'></span>

{| style="text-align: center; border: 1px solid #BBB; margin: 1em auto; max-width: 100%;" 
|-
|


[[Image:draft_Content_229468638-1-s2.0-S1110016816300047-gr2.jpg|center|534px|CFD data center physical model and different layouts of CRAC(s) units.]]


|-
| <span style="text-align: center; font-size: 75%;">

Figure 2.

CFD data center physical model and different layouts of CRAC(s) units.

</span>
|}

A typical raised floor plenum of depth 0.6 m and perforated tiles of 0.6 × 0.6 m size with an opening ratio of 25% are used for all the analyses. The cold aisle accommodates fourteen perforated tiles to provide the supply cold air to the fourteen racks. All tiles are assumed to give the air flow rate at the same temperature (12 °C) to all racks. The hot air discharged from the rack servers flows out from the room through ceiling vents. Six ceiling air outlets (three in each hot aisle between each row of rack and the room wall) are used for this purpose. Two layout arrangements (Layouts 1 and 2) of CRACs distributions in the data centers are studied. In layout 1, two CRACs units are used, one in each side of the cold aisle tiles row. In layout 2, one CRAC unit is located in one side of the racks row. The total cooling capacity of the CRACs unit(s) in the tow arrangements are the same. The CRACs units discharge the cold air at 12 °C to the plenum underneath the raised floor. The data center CFD modeling firstly modeled the distribution of air flow in raised floor plenum and the tiles. Secondly, the data center room starting from the perforated tiles was modeled. The velocity components in perforated tiles needed in the data center room model are exported from the plenum and tiles modeling. [[#f0010|Fig. 2]] gives full details of the physical model and the two layouts arrangements of the CRAC(s) units. The dashed line shown in section A-A of [[#f0010|Fig. 2]] represents cold aisle compartment that will be proposed and discussed in sections 5-2 and 5-3 for data center thermal enhancement.

To simplify the meshing details of tiles perforations, the method presented by Abdelmaksoud et al. [[#b0040|[8]]] for accurate simplification of the momentum source was utilized in the present study. The method adds a body force filed above the tiles to correct the momentum deficit of flow through a fully open tile to accurately resemble a perforated tile. The required body force was given by:

<span id='e0005'></span>
{| class="formulaSCP" style="width: 100%; text-align: center;" 
|-
| 
{| style="text-align: center; margin:auto;" 
|-
| <math>F=\frac{1}{V}\quad \rho Q\left(\frac{Q}{\sigma \quad A\quad tile}-\right. </math><math>\left. \frac{Q}{A\quad tile}\right)</math>
|}
| style="width: 5px;text-align: right;white-space: nowrap;" | (1)
|}

where ''V'' is the body force computational volume, ''ρ'' is density of air density, ''Q'' is fully open tile air flow rate, ''σ'' is the opening ratio of perforation tile and ''A<sub>tile</sub>'' is tile fully open area. A 0.534 × 0.534 × 0.01 m computational volume above each perforated tile was considered and the body force was calculated using Eq. [[#e0005|(1)]]. The flow through the server is modeled by specifying a fixed mass flow rate in and out of each server. A typically perforation factor of 0.35 was assumed for each rack. Similar to the perforated tiles, a momentum source was considered behind each server. A 0.610 × 0.500 × 0.0763 computational volume is considered for the server body force. Gambit software was used to mesh the model using tetrahedron elements. Refined mesh elements were used around servers, perforated tiles and room outlets.

==3. Mathematical formulation==

This section introduces the governing equations and boundary conditions used to simulate the fluid flow and heat transfer in a data center thermal model. In order to model the flow in a domain on the scale of a data center, turbulent transport must be included. Since the readers have some background in the field, a complete derivation of the equations will not be included. The resulting coupled partial differential equations with complex boundary conditions are solved numerically using the finite volume discretization method in the three-dimensional data center computational domains.

===3.1. Governing equations and numerical solution techniques===

Mass, momentum and energy conservation equation applied on infinitesimal fixed control volume are derived. In order to model the flow in a domain on the scale of a data center, turbulent transport is included. The obtained nonlinear coupled equations are solved in order to determine the flow velocity, temperature and pressure distributions in space and time. These equations are typically solved by using numerical techniques for most problems of any significant complexity. These techniques typically reduce the differential to algebraic equations to be solved in the flow filed using grid meshes. Because of the low Mach number (≪0.3) nature of the data center flow environment, the air can be treated as incompressible flow; in addition, the effects of natural convection are not considered. In data centers, the air enters the plenum from the CRAC and circulates through the plenum and the room volume and returns to the CRAC so that the air volume is constant.

The conservation of momentum for an incompressible, Newtonian fluid is applied. The effect of turbulent transport must be accounted for using the Reynolds averaged form of the governing equations. Because of the nonlinear aspect of the conservation of momentum equations, additional terms arise that correspond to the turbulent stresses (Reynolds stresses). These additional terms must be related to the averaged flow variables using a turbulence model. A straightforward approach to account for the turbulent transport in the averaged Navier–Stokes equations is to introduce a turbulent viscosity ''ν<sub>T</sub>'', which is not a constant like the fluid viscosity but must be determined in terms of the averaged flow variables  [[#b0120|[24]]].

The conservation of energy law comes from the first law of thermodynamics for a control volume and the results in a transport equation for the energy of the system. The thermal energy equation must also be modified to account for turbulent energy transport in a manner similar to the momentum equations. Although a data center model of the most accurate detail will include heat source at the server level, for the simplicity of this work, a black-box approach of fixed heat dissipation will be used for each rack. As discussed previously, studies have shown that the details of the server racks do not contribute very much to the entire data center model.

The system of conservation equations is solved numerically under the boundary conditions of the problem. [[#t0005|Table 1]] gives the boundary conditions for the CFD modeling of data center. The server air flow rate is estimated from the server heat generation and allowable predefined air temperature difference across the server (10.19 °C [[#b0010|[2]]]). The simulation was run using CFD-Fluent-Solver using k-Epsilon turbulence model. A 0.001 convergence criterion of the root mean square of the equation residuals was set. The results were processed and analyzed using CFD-Post. The two commercial CFD packages ANSYS FLUENT 14 and grid-generation software GAMBIT 2.4.6 are used for data processing, analysis and presentation.

<span id='t0005'></span>

{| class="wikitable" style="min-width: 60%;margin-left: auto; margin-right: auto;"
|+

Table 1.

Data center-CFD model boundary conditions.

|-

! Boundary condition
! Symbol
! Equation
! Value
|-

| Inlet airflow velocity
| ''U''<sub>0</sub>
| –
| 1 ms<sup>−1</sup>
|-

| Tile flow rate
| ''Q<sub>t</sub>''
| ''Q<sub>t</sub>'' = ''U''<sub>0</sub>''A<sub>tile</sub>''
| 0.294 m<sup>3</sup> s<sup>−1</sup>
|-

| Inlet air temperature
| ''T''<sub>0</sub>
| –
| 12 °C
|-

| Server flow rate
| ''Q<sub>s</sub>''
| ''Q<sub>s</sub>'' = ''Q<sub>t</sub>''/4
| 0.0735 m<sup>3</sup> s<sup>−1</sup>
|-

| Server heat generation
| ''P<sub>s</sub>''
| ''P<sub>s</sub>'' = ''ρ c<sub>p</sub> Q<sub>s</sub>'' Δ''T''
| 875 W
|-

| Tile porosity
| ''σ<sub>t</sub>''
| –
| 25%
|-

| Rack porosity
| ''σ<sub>r</sub>''
| –
| 35%
|}

===3.2. Data center performance parameters indices===

Several non-dimensional key parameters were used in the literature as scalable indices of performance parameters for data centers. The key parameter of data center cooling performance is the temperature distribution. Data center elements such as server, rack, perforated tile, and CRAC unit are thermally characterized by its intake and discharge temperatures. All the performance parameters are based these temperatures and a reference temperature which is normally the supply temperature from the CRAC (''T<sub>ref</sub>''.) Due to heat gain in raised floor and infiltration and recirculation of hot air through the racks and hot aisles (see  [[#f0005|Fig. 1]]), the rack intake air temperature becomes higher than this reference temperature. Similarly, due to infiltration, short circuiting and bypass of cool air from the tiles and cold aisles, the return air temperature to the CRAC becomes lower than the exhaust air temperature from the rack. The data center performance indices are used to evaluate and quantify cold air bypass and short circuit, hot air recirculation and energy efficiency in data centers. Enhancement of data center performance means minimize hot air recirculation, cold air bypass, and cold air short-circuiting to CRAC inlet. Three key indices: rack cooling index (RCI), supply and return heat indices (SHI & RHI), and return temperature index (RTI) are used in the literature as indices of data centers performance.

The work in which RCI was firstly introduced was conducted by Herrlin [[#b0125|[25]]]. This index relates the rack inlet temperatures with its allowable and recommended ranges for reliable facility continuous operation [[#b0130|[26]]], see [[#f0015|Fig. 3]]. Over-temperature means that the server intake temperature exceeds the maximum recommended one. The total over-temperature is the summation of over-temperatures of all rack servers. Similarly, under-temperature means that the intake temperatures becomes below the minimum recommended. The recommended limits are normally given by codes and guideline [[#b0125|[25]]] and [[#b0130|[26]]]. Two RCIs, namely RCI<sub>HI</sub> and RCI<sub>LO</sub>, were used to measure the server’s intake temperatures with respect to these limits. RCI<sub>HI</sub> and RCI<sub>LO</sub> measure the over and under-temperatures, respectively and can be defined by [[#b0125|[25]]]:

{| class="formulaSCP" style="width: 100%; text-align: center;" 
|-
| 
{| style="text-align: center; margin:auto;" 
|-
| <math>{\mbox{RCI}}_{\mbox{HI}}=\left[1-\frac{Total\quad over\quad temperature}{Maxallawable\quad over\quad temperature}\right]100\%</math>
|}
| style="width: 5px;text-align: right;white-space: nowrap;" | (2)
|}

{| class="formulaSCP" style="width: 100%; text-align: center;" 
|-
| 
{| style="text-align: center; margin:auto;" 
|-
| <math>{\mbox{RCI}}_{\mbox{Lo}}=\left[1-\frac{Total\quad under\quad temperature}{Maxallawable\quad under\quad temperature}\right]100\%</math>
|}
| style="width: 5px;text-align: right;white-space: nowrap;" | (3)
|}

<span id='f0015'></span>

{| style="text-align: center; border: 1px solid #BBB; margin: 1em auto; max-width: 100%;" 
|-
|


[[Image:draft_Content_229468638-1-s2.0-S1110016816300047-gr3.jpg|center|377px|Definition of total over-temperature and total under temperature [26].]]


|-
| <span style="text-align: center; font-size: 75%;">

Figure 3.

Definition of total over-temperature and total under temperature [[#b0130|[26]]].

</span>
|}

100% RCI<sub>HI</sub> is considered the ideal case where no over-temperatures exist at any rack. The lower the RCI<sub>HI</sub> means the increase in the probability of the equipment to be above maximum allowable temperature. A typical value of RCI<sub>HI</sub> in the range 91–96% can be considered as acceptable range of operation however a value below 91% is considered as a poor range of operation [[#b0125|[25]]]. The RCI<sub>LO</sub> can be considered as the complement to RCI<sub>HI</sub>, especially, when the supply condition is below the minimum recommended temperature. Different values of the maximum allowable over and under temperatures are defined by different data centers guideline and standard. Therefore these indices are a relative measure to the applied standard and guideline.

RCI<sub>HI</sub> may be enhanced by supplying more airflow rate and decreasing the supply air temperature but doing this needs more cooling energy. Therefore, the RTI index is used to measure the energy performance of data center air management system. RTI is defined by [[#b0135|[27]]]:

{| class="formulaSCP" style="width: 100%; text-align: center;" 
|-
| 
{| style="text-align: center; margin:auto;" 
|-
| <math>\mbox{RTI}=\left[\frac{T_{return}-T_{supply}}{\Delta T_{equipment}}\right]\times 100\%</math>
|}
| style="width: 5px;text-align: right;white-space: nowrap;" | (4)
|}

The RTI can also be defined as the ratio of CRACs airflow rate to the racks airflow rate. RTI above 100% indicates the existence of hot air re-circulation and the rise of server’s intake temperatures. RTI below 100% indicates the existence of cold air by-pass around the racks and go back directly to the CRAC reducing its return temperature. This almost occurs in case of increasing supply airflow to eliminate server’s hotspots. A 100 % is the target of RTI for efficient energy performance.

The supply heat index (SHI) is defined as the ratio of the heat gained by the cold air before entering the racks to the total heat gain by the air after leaving the rack. SHI can be expressed in terms of rack inlet, rack outlet and CRAC outlet temperatures as follows [[#b0125|[25]]]:

<span id='e0025'></span>
{| class="formulaSCP" style="width: 100%; text-align: center;" 
|-
| 
{| style="text-align: center; margin:auto;" 
|-
| <math>\mbox{SHI}=\left\{\frac{{\sum }_j\quad {\sum }_i\quad ((T_{in}^r)i\mbox{,}j-T_{ref})}{{\sum }_j\quad {\sum }_i\quad ((T_{out}^r)i\mbox{,}j-T_{ref})}\right\}</math>
|}
| style="width: 5px;text-align: right;white-space: nowrap;" | (5)
|}

The return heat index (RHI) is defined by:

<span id='e0030'></span>
{| class="formulaSCP" style="width: 100%; text-align: center;" 
|-
| 
{| style="text-align: center; margin:auto;" 
|-
| <math>\mbox{RHI}=\left\{\frac{{\sum }_j\quad {\sum }_i\quad M_kC_p((T_{in}^c)k-T_{ref})}{{\sum }_j\quad {\sum }_i\quad m_{i\mbox{,}j}^rC_p((T_{out}^r)i\mbox{,}j-T_{ref})}\right\}=</math><math>\frac{Total\quad heat\quad extraction\quad by\quad the\quad CRA\quad Cunits}{Total\quad Enthalpy\quad rise\quad at\quad the\quad rack\quad exhaust}</math>
|}
| style="width: 5px;text-align: right;white-space: nowrap;" | (6)
|}

RHI can be considered as a complement to SHI as it is clear from Eqs. [[#e0025|(5)]] and [[#e0030|(6)]] that

{| class="formulaSCP" style="width: 100%; text-align: center;" 
|-
| 
{| style="text-align: center; margin:auto;" 
|-
| <math>\mbox{SHI}+\mbox{RHI}=1</math>
|}
| style="width: 5px;text-align: right;white-space: nowrap;" | (7)
|}

==4. Grid Independent study and model validations==

Grid refinement study has been conducted to reach the minimum grid size that does not affect the numerical solution and insure grid independent results. The overall performance parameters indices (SHI, RHI and RTI) were calculated from the results of different meshes sizes to find the grid independent size. Results of grid independent study are illustrated in [[#t0010|Table 2]].

<span id='t0010'></span>

{| class="wikitable" style="min-width: 60%;margin-left: auto; margin-right: auto;"
|+

Table 2.

Effect of different grid size on measureable overall performance parameters (SHI, RHI and RTI).

|-

! Mesh
! Right rack RTI
! Left rack RTI
! Right rack SHI
! Left rack SHI
! Right rack RHI
! Left rack RHI
|-

| 3,220,343
| 121
| 121
| 0.2
| 0.19
| 0.8
| 0.81
|-

| 3,593,599
| 121
| 121
| 0.2
| 0.19
| 0.8
| 0.81
|-

| 4,231,343
| 121
| 121
| 0.2
| 0.19
| 0.8
| 0.81
|}

As shown in the table 3220343 mesh grid size is used.

To validate the present numerical model, the results were compared with the results of the numerical work of Fernando [[#b0140|[28]]]. The temperature distribution obtained from the present simulation was analyzed and compared with the values obtained by Fernando [[#b0115|[23]]] for 40 different points on an ''XY'' plane above the server’s tops in each rack as shown in  [[#f0020|Fig. 4]]. The temperature distribution comparison between the model and the Fernando [[#b0140|[28]]] results are illustrated in [[#f0025|Fig. 5]]. As shown in the figure, a fair agreement between the present model and Fernando measurements [[#b0140|[28]]] exists.

<span id='f0020'></span>

{| style="text-align: center; border: 1px solid #BBB; margin: 1em auto; max-width: 100%;" 
|-
|


[[Image:draft_Content_229468638-1-s2.0-S1110016816300047-gr4.jpg|center|355px|Temperature measurement points in the XY plane.]]


|-
| <span style="text-align: center; font-size: 75%;">

Figure 4.

Temperature measurement points in the ''XY'' plane.

</span>
|}

<span id='f0025'></span>

{| style="text-align: center; border: 1px solid #BBB; margin: 1em auto; max-width: 100%;" 
|-
|


[[Image:draft_Content_229468638-1-s2.0-S1110016816300047-gr5.jpg|center|357px|Comparison between present results and Fernando [28] for temperature measurement ...]]


|-
| <span style="text-align: center; font-size: 75%;">

Figure 5.

Comparison between present results and Fernando [[#b0140|[28]]] for temperature measurement points on lines 1 and 2.

</span>
|}

==5. Results and discussion==

Numerical experiments were conducted for different layouts arrangements of CRACs units distributions and different configurations of cold aisle containments. In each experiment, air flow and temperature distributions in the data center are obtained and analyzed. The data center performance index matrices SHI, RHI, RTI and RCI are calculated from the temperature distribution results. In the following sections the effects of CRACs layouts arrangements and the different configurations of cold aisle containments on the flow and temperature distribution and data center performance indices are analyzed and discussed.

===5.1. Effect of CRACs layouts arrangements===

The distribution and uniformity of the perforated tiles air flow is controlled by several factors such as plenum depth, size of flow obstructions such as cables and pipes in plenum space, layout of CRAC units with respect to the racks, tiles layout, size, and opening area and flow resistance. These factors affect the pressure distribution in the plenum space and control the flow through the perforated tiles of the data center. The CRACs layout and their locations with respect to the racks and perforated tiles can be considered as one of the data center design parameters that may affect the thermal management of the data center. Two CRACs layouts are studied; in the first layout (layout 1) the CRACs are equal distributed on the two sides of the racks rows while in the second layout (layout 2) the CRACs are located on one side of the racks rows. These are the common layouts in data centers. To compare between the two layouts, the total air flow of the CRACs is maintained the same.

[[#f0030|Fig. 6]] shows the air flow rates throw the perforated tile in front of each rack of the racks row for the two layouts. The air flow rates distribution is obtained from the modeling of the air plenum with the perforated tiles. For layout 1, the figure shows that the air flow rate is symmetric around the middle rack due to the symmetry of the problem (equal distribution of the CRACs on both sides of the racks row) where the air flow rates and consequently the air discharging velocity appears small in the tiles near the CRACs and increases until it reaches its maximum in the middle (fourth) perforated tile (PT4). This air flow distribution agrees with the results of Schmidt [[#b0025|[5]]] who showed the same trend. However, for layout 2, [[#f0030|Fig. 6]] shows the air flow rate and consequently the air velocity increases as the number of the rack in the racks row increases from PT1 to PT7.

<span id='f0030'></span>

{| style="text-align: center; border: 1px solid #BBB; margin: 1em auto; max-width: 100%;" 
|-
|


[[Image:draft_Content_229468638-1-s2.0-S1110016816300047-gr6.jpg|center|357px|Flow rates in perforated tiles for layouts 1 and 2 (kg/s).]]


|-
| <span style="text-align: center; font-size: 75%;">

Figure 6.

Flow rates in perforated tiles for layouts 1 and 2 (kg/s).

</span>
|}

[[#f0035|Fig. 7]] shows the temperature distribution around the first (PT1, PT7) and middle racks (PT4) of the racks row in layout 1 and around the first (PT1) and the last (PT7) racks of the racks rows of layout 2. As shown in the figure, a high bypass of the cold air from the cold aisle to the hot aisle occurs at the middle rack in layout 1 and at the last rack in layout 2 and high recirculation of hot air from the hot aisle to the cold aisle occurs at the first and last racks in layout 1 and at the first rack in layout 2. This hot air recirculation may lead to hot spots at the servers of these racks. This can be attributed to the air flow rate distribution along the racks row (see [[#f0035|Fig. 7]]) where increasing the supply air flow rates causes cold air bypass and decreasing the supply air flow rates causes hot air recirculation.

<span id='f0035'></span>

{| style="text-align: center; border: 1px solid #BBB; margin: 1em auto; max-width: 100%;" 
|-
|


[[Image:draft_Content_229468638-1-s2.0-S1110016816300047-gr7.jpg|center|px|Temperature distribution at the first, middle and last racks.]]


|-
| <span style="text-align: center; font-size: 75%;">

Figure 7.

Temperature distribution at the first, middle and last racks.

</span>
|}

[[#f0040|Fig. 8]] shows the distribution of the performance indices RTI, SHI and RHI of the different racks in the racks row for the two layouts of the CRACs unit. For layout 1, the figure shows the symmetry of RTI, SHI and RHI around the fourth rack due to the symmetry in air flow rates. The figure also shows the decrease in RTI and SHI and the increase in RHI of the rack as the rack number in the rack row increases until the fourth rack. This can be attributed to the increase in the air flow rate and air velocity with increasing the number of the rack in the row. Increasing the air flow rate decreases the temperature of the zone around the rack servers and increases the possibility of the cold air bypass above the rack and this decreases SHI to the recommended value  < 0.2 (good) and decreases RTI to be below the recommended value 0.92–1 (bad). The figure also shows that SHI of the middle rack lies in the recommended range (<0.2) however RTI lies below the recommended range (0.92–1). Moreover increasing air flow rates at the middle of racks and consequently the reduction of the air flow rates in the first and second racks leads to recirculation of the hot air from the hot aisle to the cold aisle at the first and second racks and this leads to the increase in RTI and SHI to be outside the recommended range (RTI becomes more than 100 % and SHI becomes more than 0.2).

<span id='f0040'></span>

{| style="text-align: center; border: 1px solid #BBB; margin: 1em auto; max-width: 100%;" 
|-
|


[[Image:draft_Content_229468638-1-s2.0-S1110016816300047-gr8.jpg|center|360px|Effect of rack numbers on RTI, SHI and RHI for layouts 1 and 2.]]


|-
| <span style="text-align: center; font-size: 75%;">

Figure 8.

Effect of rack numbers on RTI, SHI and RHI for layouts 1 and 2.

</span>
|}

For layout 2, [[#f0040|Fig. 8]] shows that RTI and SHI are very high at the first rack and then they decrease as the rack number increases until they reach minimum at the last rack. The trend of RHI is the opposite. This can be attributed to the increase in the air flow rate with increasing the rack number until it reaches maximum at the last rack. This causes high recirculation of the hot air at the first rack (RTI > 100) where the air flow rate is very low and the high cold air bypass at the last racks where the air flow rate is very high.

The effect of the rack number in the row on RCI is shown in [[#f0045|Fig. 9]] for the two layouts. For layout 1, [[#f0045|Fig. 9]] shows that RCI<sub>HI</sub> for the first rack is less than 80% (poor) and it increases with increasing the row number reaching 95% at the second rack (acceptable) and reaching its upper limit (100%) at the middle rack. On the other hand, RCI<sub>LO</sub> parameter at the first rack is 100% and it decreases as the rack number increases until it becomes less than 70% (poor) at the middle rack. These trends of RCI<sub>HI</sub> and RCI<sub>LO</sub> in layout 1 can be attributed to that the lower air flow rates at the first rack causes more recirculation of hot air from the hot aisle to the cold aisle and this increases the rack inlet temperature which leads to lower RCI<sub>HI</sub>. Increasing the air flow rate by increasing the row number eliminates this recirculation of the hot air. At the middle rack where the air flow rate becomes very high, the rack does not have any over temperature at the rack inlet (RCI<sub>HI</sub> = 1) and more bypassed cold air from the cold aisle to the hot aisle is expected.

<span id='f0045'></span>

{| style="text-align: center; border: 1px solid #BBB; margin: 1em auto; max-width: 100%;" 
|-
|


[[Image:draft_Content_229468638-1-s2.0-S1110016816300047-gr9.jpg|center|357px|Effect of rack numbers on RCI for layouts 1 and 2.]]


|-
| <span style="text-align: center; font-size: 75%;">

Figure 9.

Effect of rack numbers on RCI for layouts 1 and 2.

</span>
|}

For layout 2, [[#f0045|Fig. 9]] shows that the low tiles air flow rates at the first rack and the high tiles flow rates at the last racks adversely affect RCI parameters. This can be attributed to the hot air recirculation at the first rack and the cold air bypass at the last racks, respectively. The low air flow rate at the first rack increases the inlet air temperature, hence the RCI<sub>HI</sub> drops to 55% which is a very bad condition. The high air flow rate at the last rack decreases the inlet air temperature, hence RCI<sub>LO</sub> drops to 75% which is also very bad.

The trends of RTI, SHI, RHI, RCI <sub>HI</sub> and RCI <sub>Lo</sub> that are shown in [[#f0040|Figure 8]] and [[#f0045|Figure 9]] are supported by the temperature distribution around the first, middle and last rack that are shown in [[#f0035|Fig. 7]], where hot air recirculation at the first rack and last racks and cold air bypass at the middle rack are noticed for layout 1, and hot air recirculation at the first rack and cold air bypass at the last racks are noticed for layout 2.

===5.2. Enhancement of thermal management of data center of layout 1 using top cold aisle containment===

In the previous section it was observed for layout 1 that the low tiles air flow rates at the first and last racks in the row and the high tiles air flow rates at the middle rack cause cold air bypass at the middle rack and hot air recirculation at the first and last racks and this adversely affects RTI and SHI. These problems can be solved by adding a roof top containment (a cover plate between the top surfaces of the racks row along the entire cold aisle) as shown by the dashed line in [[#f0005|Fig. 1]].

[[#f0050|Fig. 10]] shows the temperature distribution around the first and middle racks after putting the cold aisle containment. Comparing these temperature distributions with the ones shown in [[#f0035|Fig. 7]]-a (without top containment) reveals that putting cold air containment eliminates the cold air bypass at the middle rack and the hot air recirculation at the first and last racks.

<span id='f0050'></span>

{| style="text-align: center; border: 1px solid #BBB; margin: 1em auto; max-width: 100%;" 
|-
|


[[Image:draft_Content_229468638-1-s2.0-S1110016816300047-gr10.jpg|center|px|Temperature distribution at the first and middle racks of layout 1 with roof top ...]]


|-
| <span style="text-align: center; font-size: 75%;">

Figure 10.

Temperature distribution at the first and middle racks of layout 1 with roof top containment.

</span>
|}

[[#f0055|Figure 11]] and [[#f0060|Figure 12]] illustrate the comparison between the indices parameters RTI, SHI, RHI and RCI of the two cases of layout 1; with and without top roof containment. As shown in the figures, adding roof top containment at the cold aisle improves the RTI, SHI, RHI and RCI; where both of RTI and SHI decrease to be within the recommended range about 110% and 0.2, respectively and RCI<sub>HI</sub> parameter increases to be within the recommended range about 95 % (acceptable). This can be attributed to the elimination of both hot air recirculation from the hot aisle to the cold aisle at the first and last racks and the cold air bypass from the cold aisle to the hot aisle at the middle rack.

<span id='f0055'></span>

{| style="text-align: center; border: 1px solid #BBB; margin: 1em auto; max-width: 100%;" 
|-
|


[[Image:draft_Content_229468638-1-s2.0-S1110016816300047-gr11.jpg|center|359px|Effect of adding roof top containment of the cold aisle of layout 1 on the RTI, ...]]


|-
| <span style="text-align: center; font-size: 75%;">

Figure 11.

Effect of adding roof top containment of the cold aisle of layout 1 on the RTI, SHI and RHI parameter.

</span>
|}

<span id='f0060'></span>

{| style="text-align: center; border: 1px solid #BBB; margin: 1em auto; max-width: 100%;" 
|-
|


[[Image:draft_Content_229468638-1-s2.0-S1110016816300047-gr12.jpg|center|356px|Effect of adding roof top containment of the cold aisle of layout 1 on the RCI ...]]


|-
| <span style="text-align: center; font-size: 75%;">

Figure 12.

Effect of adding roof top containment of the cold aisle of layout 1 on the RCI parameter.

</span>
|}

===5.3. Enhancement of thermal management of data center of layout 2 using top and side containments for cold aisle===

In the previous section it was observed that when the CRACs units are located on one side of the racks row, the low tiles air flow rates at the first rack in the row and the high tiles flow rates at the last racks (see [[#f0015|Fig. 3]]) cause cold air bypass at the last racks and hot air recirculation at the first rack and this adversely affects RTI, SHI and RCI. These problems can be solved by adding a roof top (a cover plate between the top surfaces of the racks row along the entire cold aisle) and side’s containments (the end sides of the two rows are closed by cover plates) of the cold aisle. Two arrangements are suggested and investigated. The first arrangement (arrangement I) uses side containment at the side of the last rack in the row and the second arrangement (arrangement II) uses top containments above the last four racks in addition to the side containment used in arrangement I.

[[#f0065|Fig. 13]] illustrates the temperature distribution of the three cases, without containment, arrangement I and arrangement II. As shown in the figure, adding side and roof top containments at the cold aisle eliminates the hot air recirculation from the hot aisle to the cold aisle at the first and second racks and the cold air bypass from the cold aisle to the hot aisle at the middle rack. The figure also shows that the enhancement of arrangements II is much better than that arrangement I

<span id='f0065'></span>

{| style="text-align: center; border: 1px solid #BBB; margin: 1em auto; max-width: 100%;" 
|-
|


[[Image:draft_Content_229468638-1-s2.0-S1110016816300047-gr13.jpg|center|px|Temperature distribution around the first and last racks for layout 2 with and ...]]


|-
| <span style="text-align: center; font-size: 75%;">

Figure 13.

Temperature distribution around the first and last racks for layout 2 with and without cold aisle containments.

</span>
|}

[[#f0070|Fig. 14]] illustrates the comparison between the indices parameters RTI, SHI and RHI of the two arrangement cases with the basic case of layout 2 (without containments). As shown in the figure, adding side containment at the last rack of the cold aisle (arrangement I) causes slight changes in RTI, RHI and SHI but cannot decrease the recirculation and bypass of air in the data center. However using the side and top containments (arrangement II) improves the RTI, SHI and RHI; where both of RTI and SHI decreased to be within the recommended range about 100 % and 0.2, respectively. This can be attributed to the complete elimination of the hot air recirculation from the hot aisle to the cold aisle at the first and second racks and the cold air bypass from the cold aisle to the hot aisle at the last racks.

<span id='f0070'></span>

{| style="text-align: center; border: 1px solid #BBB; margin: 1em auto; max-width: 100%;" 
|-
|


[[Image:draft_Content_229468638-1-s2.0-S1110016816300047-gr14.jpg|center|357px|Effect of using cold aisle containments on RTI, SHI and RHI for layout 2.]]


|-
| <span style="text-align: center; font-size: 75%;">

Figure 14.

Effect of using cold aisle containments on RTI, SHI and RHI for layout 2.

</span>
|}

[[#f0075|Fig. 15]] illustrates the comparison between the indices parameters RCI of the two arrangement cases with the basic case of layout 1 (without containments). As shown in the figure, adding side containment at the last rack of the cold aisle (arrangement I) causes slight increases in RTI<sub>HI</sub> at the first rack but it does not affect RTI<sub>LO</sub> at the last rack. However using the side and top containments (arrangement II) improves both of RTI<sub>HI</sub> at the first rack and RTI<sub>LO</sub> at the last rack but their values are still a little bit away from the recommended ranges.

<span id='f0075'></span>

{| style="text-align: center; border: 1px solid #BBB; margin: 1em auto; max-width: 100%;" 
|-
|


[[Image:draft_Content_229468638-1-s2.0-S1110016816300047-gr15.jpg|center|355px|Effect of using cold aisle containments on RCIHI and RCILO for layout 2.]]


|-
| <span style="text-align: center; font-size: 75%;">

Figure 15.

Effect of using cold aisle containments on RCI<sub>HI</sub> and RCI<sub>LO</sub> for layout 2.

</span>
|}

==6. Conclusions==

Numerical investigations of the performance of air cooled up-flow data centers using perforated air tiles have been carried out for different arrangements of racks and CRAC layouts and different configurations of side and roof top cold aisle containments. Temperature distribution, air flow characteristics particularly air recirculation and bypass and thermal managements in data centers are evaluated in terms of the measureable overall performance parameters (RTI, SHI, RHI and RCI). The results showed that:
* The hot air recirculation, cold air bypass and the measurable performance parameters of a rack strongly depend on the rack location in the racks array.
* The CRACs unit layout with respect to the racks row affects the racks thermal managements especially the first, middle and last racks in the racks rows.
* Using cold aisle containments in the different CRACs layouts decreases the recirculation and bypass of air flow around the first, middle and last racks in a rack row and improves the thermal performance of the data center as the performance indices moves to be within the recommended ranges.
* Roof top containment is more effective in thermal management enhancement as compared to side containment where the performance indices move to be within the recommended ranges.

==References==

<ol style='list-style-type: none;margin-left: 0px;'><li><span id='b0005'></span>
[[#b0005|[1]]] J.G. Koomey; Worldwide electricity used in data centers; Environ. Res. Lett., 3 (3) (2008)</li>
<li><span id='b0010'></span>
[[#b0010|[2]]] Emerson Network Power, Energy logic: Reducing Data Center Energy Consumption by Creating Savings that Cascade Across Systems, a White Paper from the Experts in Business-Critical Continuity, 2008.</li>
<li><span id='b0015'></span>
[[#b0015|[3]]] J. Cho, T. Lim, B.S. Kim; Measurements and predictions of the air distribution systems in high compute density (internet) data centers; Energy Build., 41 (10) (2009), pp. 1107–1115</li>
<li><span id='b0020'></span>
[[#b0020|[4]]] S. Kang, R.R. Schmidt, K.M. Kelkar, A. Rdmehr, S.V. Patankar, A methodology for the design of perforated tiles in raised floor data centers using computational flow analysis, in: Proceedings of the Inter Society Conference on Thermal Phenomena (ITherm), 2000.</li>
<li><span id='b0025'></span>
[[#b0025|[5]]] R. Schmidt, K. Karki, K. Kelkar, A. Radmehr, S. Patankar, Measurements and predictions of the flow distribution through perforated tiles in raised floor data centers, in: Advanced in Electronic Packaging 2001, Proceedings of IPACK’01, and the Pacific Rim/ASME International Electronic Packaging Technical Conference and Exhibition in Kauai, vol. 2, Paper IPACK2001-15728, 2001b, pp. 905–914.</li>
<li><span id='b0030'></span>
[[#b0030|[6]]] Triad, Denver, Colorado, 2011. <http://triadfloors.com/tile-efficiency.php>.</li>
<li><span id='b0035'></span>
[[#b0035|[7]]] K.C. Karki, A. Radmehr, S.V. Patankar; Use of computational fluid dynamics for calculating flow rates through perforated tiles in raised-floor data centers; Int. J. Heat. Ventialat. Air-Cond. Refrig. Res., 9 (2) (2003), pp. 153–166</li>
<li><span id='b0040'></span>
[[#b0040|[8]]] W.A. Abdelmaksoud, E. Khalifa, T.Q. Dang, B. Elhadidi, R.R. Schmidt, M. Iyengar, Experimental and computational study of perforated floor tile in data centers, In: Proceedings of the Inter Society Conference on Thermal Phenomena (ITherm), Las Vegas, Nevada, USA, 2012.</li>
<li><span id='b0045'></span>
[[#b0045|[9]]] S. Kim; Effectiveness of Specialized Floor Tile Designs on Air Flow Uniformity; State University of New York at Binghamton, BS (2009)</li>
<li><span id='b0050'></span>
[[#b0050|[10]]] R.R. Schmidt, E. Cruz; Cluster of high-powered racks within a raised-floor computer data center: effect of perforated tile flow distribution on rack inlet temperatures; ASME J. Electron. Packag., 126 (2004), pp. 510–518</li>
<li><span id='b0055'></span>
[[#b0055|[11]]] J. Cho, T. Lim, B.S. Kim; Measurements and Predictions of the Air Distribution Systems in High Compute Density (Internet) Data Centers; Yonsei University, Seoul 120-749, South Korea (2009)</li>
<li><span id='b0060'></span>
[[#b0060|[12]]] R.R. Schmidt, Airflow uniformity through perforated tiles in a raised-floor data center, in: Proceedings of the ASME InterPACK ″05 Conference, San Francisco, CA, Paper IPACK 2005-73375, 2005.</li>
<li><span id='b0065'></span>
[[#b0065|[13]]] S. Bhopte, R. Schmidt, D. Agonafer, B. Sammaika; Optimization of data center room layout to minimize rack inlet air temperature; J. Electron. Packag., 128 (2006), pp. 380–387</li>
<li><span id='b0070'></span>
[[#b0070|[14]]] K.C. Karki, S.V. Patankar; Airflow Distribution Through Perforated Tiles in Raised-Floor Data Centers; Innovative Research Inc., 3025 Harbor Lane North, Suite 300, Plymouth, MN 55447, USA (2005)</li>
<li><span id='b0075'></span>
[[#b0075|[15]]] K.C. Karki, S.V. Patankar, A. Radmehr, Techniques for controlling airflow distribution in raised-floor data centers, in: Proceedings of IPACK03 The Pacific Rim/ASME International Electronic Packaging, Technical Conference and Exhibition, July 6–11, 2003, Maui, Hawaii, USA.</li>
<li><span id='b0080'></span>
[[#b0080|[16]]] R.K. Sharma; Dimensionless Parameters for Evaluation of Thermal Design and Performance of Large-Scale Data Centers; Hewlett-Packard Laboratories (2002)</li>
<li><span id='b0085'></span>
[[#b0085|[17]]] K. Zhang, X. Zhang, S. Li, X. Jin; Experimental study on the characteristics of supply air for UFAD system with perforated tiles; Energy Build., 80 (2014), pp. 1–6</li>
<li><span id='b0090'></span>
[[#b0090|[18]]] M. Ibrahim; Dynamic Thermal Management of Data Centers; Binghamton University, State University of New York (2012)</li>
<li><span id='b0095'></span>
[[#b0095|[19]]] S.A. Nada, M.A. Said, M.A. Rady; Numerical investigation and parametric study for thermal and energy management enhancements in data centers buildings; Appl. Therm. Eng., 98 (2016), pp. 110–128</li>
<li><span id='b0100'></span>
[[#b0100|[20]]] S.A. Nada, K.E. Elfeky, A.M. A Attia; Experimental investigations of air conditioning solutions in high power density data centers using a scaled physical model; Int. J. Refrig., 63 (2016), pp. 87–99</li>
<li><span id='b0105'></span>
[[#b0105|[21]]] S.A. Nada, K.E. Elfeky; Experimental investigations of thermal managements solutions in data centers buildings for different arrangements of cold aisles containments; J. Build. Eng., 5 (2016), pp. 41–49</li>
<li><span id='b0110'></span>
[[#b0110|[22]]] S.A. Nada, K.E. Elfeky, Ali M.A. Attia, W.G. Alshaer; Thermal management of electronic servers under different power conditions; Int. J. Emerg. Trends Electr. Electr. (IJETEE), 11 (4) (2015), pp. 145–150</li>
<li><span id='b0115'></span>
[[#b0115|[23]]] S.A. Nada, M.A. Rady, M. Elsharnoby, M.A. Said; Numerical investigation of cooling of electronic servers racks at different locations and spacing from the data center cooling unit; Int. J. Curr. Eng. Technol., 5 (5) (2015), pp. 3448–3456</li>
<li><span id='b0120'></span>
[[#b0120|[24]]] J.C. Tannehill, D.A. Anderson, R.H. Pletcher; Computational Fluid Mechanics and Heat Transfer;  (second ed.)Taylor & Francis, Washington, DC (1997)</li>
<li><span id='b0125'></span>
[[#b0125|[25]]] M.K. Herrlin; Rack cooling effectiveness in data centers and telecom central offices: the rack cooling index (RCI); ASHRAE Trans., 111 (Part 2) (2005) (American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc., Atlanta, GA)</li>
<li><span id='b0130'></span>
[[#b0130|[26]]] TC 9.9, Mission Critical Facilities, Technology Spaces, and Electronic Equipment, Thermal Guidelines for Data Processing Environments, American Society of Heating, Refrigerating and Air-Conditioning Engineers Inc., 2008.</li>
<li><span id='b0135'></span>
[[#b0135|[27]]] J. Cho, B.S. Kim; Evaluation of air management systems thermal performance for superior cooling efficiency in high-density data centers; Energy Build., 43 (2011), pp. 2145–2155</li>
<li><span id='b0140'></span>
[[#b0140|[28]]] H. Fernando; Can a Data Center Heat-Flow Model be Scaled Down?; The University of Melbourne, Parkville, Melbourne, Australia (2012)</li>
</ol>