System
Choosing the right system platform is as critical as choosing the right processor and interconnects for your High Performance cluster. The busing and layout of your cluster nodes can have profound implications in the performance of your cluster. Today’s point-to-point interconnections such as Intel’s’ QuickPath and AMD’s HyperTransport allow for near linear scaling of sockets with respect to memory bandwidth, while minimizing latencies.
Today’s I/O Hub or Media and communication processors can have many options such as their PCI Express (PCIe) or PCI-X configuration, SAS/SATA/SCSI/RAID options, and add-ons along with management options such as Intelligent Platform Management Interface (IPMI) with or without KVM over LAN.
PCIe configurations
PCIe configurations can be one of the most confusing parts of choosing the correct system platform for your cluster. There are three options with PCIe that you have to consider:
- The Generation or Version of PCIe used
- The physical connector used
- The number of lanes connected inside the connector
PCIe was created by Intel in 2004 as a standard to replace the older PCI, PCI-X, and AGP standards. PCIe uses a point to point serial link rather than older parallel bus technologies. PCIe v1.x has 2.5GT/s per link bandwidth translating to 250MB/s per link. The standard was updated in early 2007 doubling the per-link bandwidth to 5.0 GT/s or 500 MB/s per link. Generally speaking, it is better to use PCIe v2.0 whenever possible.
PCIe can have five different connectors, they are x1, x4 , x8, x16, x32. The use of x32 connectors is extremely rare. The x8 nomenclature means that the physical connector can handle up to 8 PCIe lanes, but does not mean that 8 lanes are connected. This makes it possible to have a physical x16 connector but have only 1, 4 or 8 lanes connected. The speed of the connection is limited to only the number of lanes connected. For example a x16 connector with a x4 link, is limited to the bandwidth of a x4 link or 1GB/s.
The number of links used with a card should be the main issue at hand when selecting a system platform. Most Interconnect cards use a x8 link like a Inifiniband, Mynirnet, or 10G/1G Ethernet cards. Most co-processors like a nVidia Telsa, AMD FireStream, Xilinix FPGA , or Cell microprocessors use a x16 link for best performance.
PCI-X Configuration
Most PCI-X configurations use a bridge to convert from PCIe to PCI-X. While ideally just using a PCIe card is preferred, a PCI-X chip may be required for compatibility and service reasons.
PCI-X comes in two versions, PCI-X 133 and PCI-X 100. Due to the parallel nature of PCI-X, most bridge chips today offer PCI-X 133 throughput when connected to a single card and PCI-X 100 throughput when using 2 or 3 cards.
SAS/SATA/SCSI options
Nodes in clusters range from being diskless to having full-blown RAID arrays. Most system boards inside a cluster have options for onboard/integrated SAS, SCSI and RAID. When considering your options is is important to consider two things: access time and throughput. Drives that have faster spindle speeds, like SAS or SCSI, should be used when access time has a dramatic effect in the performance of your applications. RAID arrays should be used when a single disk can no longer meet your needs. Reasons for this can include reliability, or more throughput (Note: this does not decrease access times). Most on-board RAID setups requires a good amount of overhead and are generally not recommended.
Remote Management
The Intelligent Platform Management Interface (IPMI) is a standard to monitor system health and manage systems or nodes in a cluster. IPMI operates independently of the operating system and allows for management of a system remotely even in the absence of an operating system, system management software, or even if the monitored system is powered off (but connected to a power source). Aspen Systems has a full suite of IMPI cards that can offer features such as direct serial connection, a local area network (LAN), serial over LAN (SOL) connection, and KVM over LAN. When installed with our ABC Cluster Management Suite, your cluster can be managed remotely from any SSL enabled web browser. IPMI management permits monitoring of each node – temperature, fan, processor utilization, drive status, and much more.
Since choosing your system platform can be quite overwhelming, contact your sales engineer today to discuss your options.
