Ethernet

Gigabit networking has evolved as the dominant standard in networking. The result of research done at Xerox Corporation in the early 1970s, Ethernet has evolved into the most widely deployed network technology today. Gigabit Ethernet was standardized by the IEEE in June 1998 as IEEE 802.3z and in 1999, IEEE 802.3ab was ratified as Standardizing Gigabit Ethernet over category 5e, or 6 cabling and became known as 1000BASE-T. In November 2007, 54% of the TOP500 Supercomputers used Gigabit Ethernet.

Since that time, faster 10 Gigabit Ethernet standards have become available as the IEEE ratified a fiber-based standard in 2002, and a twisted pair standard (CAT 6) in 2006. 10-Gigabit Ethernet products are available, but may not be optimal for price-performance systems due to low port densities and cost per port.

Gigabit or 10 Gigabit Ethernet can also be used to interconnect your HPC cluster providing latencies ranging from ~40us to ~300us. However, many codes require higher bandwidth or lower latency than standard Gigabit Ethernet interfaces and switches to operate efficiently. Both InfiniBand and Myrinet technologies are commonly used in these cases.

Some clusters utilize Ethernet bonding, which bonds two Gigabit or 10 Gigabit Ethernet interfaces together to provide more bandwidth than a single Gigabit Ethernet interface can provide. Your switch and your distribution must provide/support this capability. Bonding is generally not recommended as this can interfere with some MPI implementations.