Configuring a server for optimal performance

The preceding posts have illustrated the major building blocks that effect server configuration; I explained the importance of each one and the priority of adding it to the system.
If you missed them you can check these links:

• Introduction to DDR3
• DIMM Ranks and configuration
• Intel Xeon 5600 (Westmere) family
• Populating DIMMs considerations, Order and Ranks

In this final post of server configurations, I will present examples of configurations and areas where they should be applied.

Major Configurations
The configuration of a server is derived from its target application requirements. There are four major configurations

1. Maximum Performance 
2. Balanced Performance 
3. Maximum Capacity 
4. RAS configurations

Maximum Performance 

This configuration is intended to get the maximum CPU frequency, and maximum memory bandwidth. It usually uses low count of memory, as you populate only one DIMM per channel (i.e 6 DIMMS overall). The common use for such servers is for High Performance Computing ( HPC) in research organization, Oil & Gas industry and Chip Design.  

 Figure 1:  Maximum Performance

Best configuration at the time of publishing this post:

• CPU - Intel Xeon X5680 (3.33GHz), 6 cores per processor.
  
• Memory - 6 PC3-10600 DIMMS (such as Kingston KVR1333D3D4R9SK3/24G) to allow 48GB of RAM, at 10.6GB/s bandwidth to memory.

  Balanced Performance 

This configuration is focused on getting a balanced configuration between the maximum CPU frequency, and maximum capacity of memory. It usually uses medium count of memory, up to 96GB per host. The common use for such servers is for virtualization and other standard enterprise applications.  

 Figure 2:  Balanced Performance

Best configuration at the time of publishing this post:

• CPU - Intel Xeon X5680 (3.33GHz), 6 cores per processor.
  
• Memory - 2 DPC, 12 PC3-8500 DIMMS (such as Kingston KVR1066D3Q8R7SK3/24G) to allow 96GB of RAM, at 8.5GB/s bandwidth to memory. 

  Maximum Capacity

This configuration is focused on getting a configuration that will support the maximum capacity of memory, with a considerable compute power. It usually designed to use as much as 144GB of RAM per host  ( 296GB with the upcoming 16GB modules). The common use for such servers is for very large scale database servers.  

 Figure 3:  Maximum Capacity

Best configuration at the time of publishing this post:

• CPU - Intel Xeon X5680 (3.33GHz), 6 cores per processor.
  
• Memory - 3 DPC, 18 PC3-8500 DIMMS (such as Kingston KVR1066D3Q8R7SK3/24G) to allow 144GB of RAM, at 6.4GB/s bandwidth to memory. 

 RAS Configuration

RAS stands for Reliability, Availability and Serviceability.  Although the ECC technology offers error correction, it does not provide any failover capability. Replacing a DIMM in case of failure requires a power down of the system. The RAS configurations offer three memory protection options:

1. Online spare memory mode
2. Mirrored memory mode
3. Lockstep memory mode

 

          This configuration uses only two out the three channels.

 Figure 4:  RAS configuration

   Online spare memory mode

In this mode, one of the channels is designed as spare. This channel is not used in normal system operation. If a working DIMM exceeds the threshold of correctable memory errors, the system switches to the standby channel and the faulty channel is taken offline. 

 

   Mirrored memory mode

In this mode, the same data is written to each channel and the read is alternated between the two channels. If a working DIMM exceeds the threshold of correctable memory errors in one of the channels, the faulty channel is taken offline and the system switches to using only one channel. 

 

   Lockstep memory mode

This mode uses two memory channels at a time, and the work as a single channel. Each read and write operations moves a data word two channel wide. To provide double 8-bit error correction within a single DRAM. This mode is the most reliable but it reduces the maximum memory capacity as the third channel is not used.

Summary
By now you should have the tools to configure your server for the optimal performance you will need for your application. You should focus on the application's memory requirements and start from that point to configure how much memory you should use and in which configuration of ranking and population.

-Partykof

Populating DIMMs considerations, Order and Ranks

The Nehalem and Westmere platforms offer a wide variety of DIMM configurations. Some of the various DIMM configurations are shown below

Feature	Values
Number of DIMMs	1,2 or 3
Number of DIMMs slots per channel	2 or 3 DIMM Slots
Number of DIMMs populated per channel	1,2 or 3 DIMM per channel
DIMM Frequencies	DDR3-800, DDR3-1066, DDR3-1300

 Table 1:  DIMM Configurations

Populating DIMMs within a channel

When populating DIMMs in a three slots per channel configurations, a “fill-farthest” approach is used, meaning, the farthest DIMM from the processor is used first. If a Quad-rank DIMM is used, it should be populated first.

Figure 1:  DIMM Population within a channel

DIMM population in an 18 DIMM slots configuration

	CPU1		CPU2
	Slot Number	Population Order	Slot Number	Population Order
Channel1	1	G	1	G
	2	D	2	D
	3	A	3	A
Channel2	4	H	4	H
	5	E	5	E
	6	B	6	B
Channel3	7	I	7	I
	8	F	8	F
	9	C	9	C

 Table 2:  DIMM Population in 18 DIMM Slots

Additional population requirements

1. All DIMMS must be DDR3 DIMMs.
2. The 5600 series support low voltage DDR3 memory (DDR3L) 1.35V, the 5500 supports only 1.5V, if mixed they will work at 1.5V.
3. Mixing Registered and Unbuffered DIMMs is not allowed.
4. The maximum supported speed is defined by the BIOS and not the DIMMs
5. Mixing different timing DIMMs will force the operation at the slowest DIMM for both processors.

RDIMM Ranks population in a three slots per channel configuration

Configuration Number	Max Speed	DIMM2	DIMM1	DIMM0
1	DDR3-1333	-	-	Single-rank
2	DDR3-1333	-	-	Dual-rank
3	DDR3-1066	-	-	Quad-rank
4	DDR3-1066	-	Single-rank	Single-rank
5	DDR3-1066	-	Single-rank	Dual-rank
6	DDR3-1066	-	Dual-rank	Single-rank
7	DDR3-1066	-	Dual-rank	Dual-rank
8	DDR3-800	-	Single-rank	Quad-rank
9	DDR3-800	-	Dual-rank	Quad-rank
10	DDR3-800	-	Quad-rank	Quad-rank
11	DDR3-800	Single-rank	Single-rank	Single-rank
12	DDR3-800	Single-rank	Single-rank	Dual-rank
13	DDR3-800	Single-rank	Dual-rank	Single-rank
14	DDR3-800	Dual-rank	Single-rank	Single-rank
15	DDR3-800	Single-rank	Dual-rank	Dual-rank
16	DDR3-800	Dual-rank	Single-rank	Dual-rank
17	DDR3-800	Dual-rank	Dual-rank	Single-rank
18	DDR3-800	Dual-rank	Dual-rank	Dual-rank

 Table 3:  DIMM RANKS Population in 3 slots per channel

This concludes all the basic elements we need for configuring the perfect server.

-Partykof

Intel Xeon 5600 (Westmere) family

In the series of posts related to server configuration I reviewed the memory configuration for optimal performance. I figured that a short introduction to the 5600 family would be useful to better understand the considerations that will follow in my next post.   

About the Xeon 5600
Part of Intel Tick-Tock cadence, The Intel Xeon 5500 (Nehalem) was shrink and replaced by Intel Xeon 5600 (Westmere) family, to be followed by Sandy Bride next year which will introduce a new microarchitecture.
The 5600 is based on the new 32nm process used in chip manufacturing and offers better performance at lower power consumption.

The 5500 and 5600 series introduced a new concept in the Xeon family, they integrate a DDR3 memory controller, that allows via 3 channels, a direct connection to a dedicated memory, up to 3 DIMMs per channel. It also includes a link controller that handles the communication with the neighbor processor in what Intel calls QuickPath interconnect (QPI). the QPI allows up to 6.4GT/s in each direction per link.

For virtualization it includes the Extended VT-x, VT-c and the VT-d technologies.
The storage interface include a 6 ports SATA2 with software raid5.

Intel introduced to these processors a new acceleration technology called, Turbo Boost Technology, which automatically allows the processor to run faster then the marked frequency in cretin conditions. The max turbo boost frequency is dependent on the number of active cores within a processor.

 Figure 1: Xeon 5600 DP architecture

The Xeon 5600 platform advantages  

• Higher clock speeds
• More cores
• More cache
• More memory
• New instructions
• Supports Low voltage DDR3 (LDDR3)

The Xeon 5600 is available in the following configurations:

Processor Model	Processor Base Frequency	Cores	L3 Cache	Power	Intel ® QPI Speed	Max Turbo Frequency	Number of Threads
X5680	3.33 GHz	6	12MB	130 W	6.4 GT / s	3.6 GHz	12
X5677	3.46 GHz	4	12MB	130 W	6.4 GT / s	3.733 GHz	8
X5670	2.93 GHz	6	12MB	95 W	6.4 GT / s	3.333 GHz	12
X5667	3.066 GHz	4	12MB	95 W	6.4 GT / s	3.466 GHz	8
X5660	2.8 GHz	6	12MB	95 W	6.4 GT / s	3.2 GHz	12
X5650	2.66 GHz	6	12MB	95 W	6.4 GT / s	3.066 GHz	12
L5640	2.26 GHz	6	12MB	60 W	5.86 GT / s	2.666 GHz	12
L5630	2.13 GHz	4	12MB	40 W	5.86 GT / s	2.4 GHz	8
L5609	1.86 GHz	4	12MB	40 W	4.8 GT / s	1.866 GHz	4
E5640	2.66 GHz	4	12MB	80 W	5.86 GT / s	2.933 GHz	8
E5630	2.53 GHz	4	12MB	80 W	5.86 GT / s	2.8 GHz	8
E5620	2.4 GHz	4	12MB	80 W	5.86 GT / s	2.666 GHz	8

Table 1: Xeon 5600 configurations

Notice the interesting low voltage processors that are available in 40 and 60 watt versions. One possible application is to use them in a passively cooled chassis (i.e. without fans). 

Now that we've got the understanding of these basic elements, it is time to put things in to practice. In my next post we'll start configuring the idle server.
 

DIMM Ranks and configuration

Server performance is highly related to its hardware configuration, In order to understand how best to configure the server a deeper understanding of its component is required. In my recent post, Introduction to DDR3 I wrote about the DDR3 technology, the different memory frequencies, 800, 1333, 1600 MHz that are adopted in recent servers CPU technology.

Another important factor to consider when configuring server memory is memory ranks, as populating a system with the wrong memory modules could result in reduction in the total capacity of the server and could create problems in performance. Some servers limit the number of memory ranks, if the number of memory ranks exceeds the specified maximum ranks set by the motherboard chipset; the server may not boot or may not operate reliably.

Memory DIMMs are configured with DRAM chips, each DRAM chip provides either 4bits (i.e x4) or 8bits ( i.e x8) of a 64bit data word. For Error Correction Code (ECC) you need 72bit, therefore 9 chips of x8 are needed to construct a 72bit data word, or 18 chips of x4.

Single-sided and double-sided DIMMs

An ECC DIMM with all nine DRAM chips on one side is called single-sided, and an ECC DIMM with nine DRAM chips on each side is called double-sided ( see Figure 1). A single-sided x8 ECC DIMM and a double-sided x4 ECC DIMM each create a single block of 72bits (64bits plus 8 ECC bits).

Single-rank, dual-rank, and quad-rank DIMMs 

DIMMs are also classified to ranks. The definition of a rank is an area or block of 64-bits (72bits for ECC memory). A single-rank ECC DIMM uses all of its DRAM chips to create a single block of 72bits, and all the chips are activated by one chip-select (CS) signal from the memory controller. A dual-rank will require two chip-select signals as it produces two blocks from two sets of chips on the DIMM. The same logic applies to Quad-rank, where you need four CS signals.

Figure 1: 72bit SDRAM DIMMs and corresponding DIMM rank

The first advantage for higher rank DIMM is the greater capacity they can offer per the available DIMM slots on the server’s motherboard. The second advantage is related to latency, where there is a higher probability for the memory controller to relocate pending memory pages.
The number of ranks per channel also effects the performance, with an odd number of ranks per channel there is performance disadvantage of realistically 2-3%. Since there 3 channels per CPU, the use dual-rank DIMMS will always result in an even number of ranks, which is another good reason why to use it.

Introduction to DDR3

Recent CPU technology has adopted the use of high speed memory module, which makes server configuration a bit tricky. I often had to figure out what kind of memory modules to use on my servers, be it a Database Server, an High Performance Computing (HPC) node or a Virtualization platform.

I hope the next series of posts will help shed some light on this matter and assist building the optimized platform for your application.

First we must understand the basics of memory technology that is currently used.  

JEDEC - Joint Electronic Device Engineering Council, is the group that set memory industry standards ,aspects such as features, functionality, AC and DC parameters, data interfaces and more.

The DDR3 SDRAM is the latest one used for servers. It is a high-speed dynamic random-access memory internally configured as an eight-bank DRAM. The DDR3 SDRAM uses a 8n prefetch architecture to achieve high-speed operation. The 8n prefetch architecture is combined with an interface designed to transfer two data words per clock cycle at the I/O pins.
Theoretically, these modules could transfer data at the effective clockrate of 800-1600MHz (for a single clock bandwidth of 400-800MHz), producing a throughput of 12800 MB/s.

Chart 1: SDRAM technologies throughput

Compatibility

DDR3 DIMMS have 240 pins, the same number as DDR2; however, since the supply voltage is different DDR3 uses 1.5V where DDR2 uses 1.8V they are incompatible. To identify between them, the DIMMS are also physically incompatible. The key notch location is different. so DO NOT try forcibly to install them. 


 DDR3 Advantages

• Higher Performance – The most important benefit is the ability to transfer 8bits of data in 1 clock cycle, which is twice than DDR2.  Higher throughput equals higher performance.
• Lower Power – DDR3 memory operate at 1.5V supply, which is 17% less than what DDR2 uses. This low power results in improving the thermal design using 30% less power.
• Larger Density – Since the DDR3 allows higher chip capacity (1GB<) it allows a maximum memory modules of 16GB.
• Data Integrity – An integrated thermal sensor refreshes the data in the device and ensures data integrity and data drainage during transfers
• Higher Speed – Twice the speed of DDR2, it has the ability to transfer I/O data eight times the speed of the memory cell it contains.

DDR3 Standard Modules

Module Name	Standard Name	Memory Clock	I/O Bus Clock	Data Rate	Peak Transfer Rate
PC3-6400	DDR3-800	100 MHz	400 MHz	800 MT/s	6400 MB/s
PC3-8500	DDR3-1066	133 MHz	533 MHz	1066 MT/s	8533 MB/s
PC3-10600	DDR3-1333	166 MHz	667 MHz	1333 MT/s	10667 MB/s
PC3-12800	DDR3-1600	200 MHz	800 MHz	1600 MT/s	12800 MB/s

Table 1: DDR3 standard memory modules

Note that Standard’s name correlates to the clockrate, not the Memory Clock or the Bus Clock, i.e DDR3-1600 offers the clockrate of 1600MHz.

Since memory configuration and capacity is effecting server performance, it is very important to understand which modules to use in which configuration. For example, populating all memory banks with the fastest modules (DDR3-1600) will cause the system to work as if DDR3-800 where used. in the next posts I’ll try to explain why.