- HPC Buyers Guide
Detailed Buyers Reference- Applications
- Budget
- Processors
- Interconnects
- Memory
- Distributions
- Cluster Management
- Warranty and Support
- Cluster Racking
- Cluster Logical Layout
- Master Node
- Storage
- Backup
- Nodes
- Cooling
- Commercial Software
- Schedulers
- Account Management
- Security
- Windows Integration
- Shipping and Delivery
- On-Site and Training
- Next Steps
HPC Applications- Processors
- System
- Network
- Storage
- Power
- Cooling
- Facilities Estimator
- Remote Management
- GPU Computing
- Green Computing
- Visualization
- Compilers
- Operating Systems
- Virtualization
- Schedulers and Tools
Power
Power Distribution Units (PDUs)
Modern clusters can require significant power. Each rack in your Aspen cluster is normally equipped with rack mounted PDUs which provide power to one or more nodes. Normally, one or more PDUs are installed in the rear of the rack behind the node mounting infrastructure, and do not impact the rack space available for mounting your other hardware. Aspen can provide your cluster with switched PDUs. ABC, or cluster administrators can use these switched PDUs to remotely power off any system in the cluster. Metered PDUs are available as well, which ABC can poll for circuit status and load.
These PDUs are usually connected directly to outlets on the wall, under your raised floor, or in your overhead rack infrastructure. They can also be connected to Un-interruptible Power Supply (UPS) units which are located in your Aspen rack(s) or elsewhere.
UPS Systems
You may have a facility that already has a UPS or even a generator. A UPS unit is necessary to insure that no power interruption occurs, even if you have a generator. A UPS unit alone will only keep the units powered for a limited amount of time, usually less than 30 minutes, but can be used in conjunction with a facility generator to insure that your infrastructure continues to run even in extended power outages.
It is always a good idea to protect your critical systems, normally the master, any fail-over masters, administrative, and storage nodes, with UPS systems if your facility does not have them. Operating these nodes on UPS ensures that a sudden power blackout (power is lost completely) or brownout (low voltage levels), or dropout (momentary total loss of power) does not cause these nodes to crash, which risks file system or hardware damage. Brownout or dropout situations are transparent to a node protected by UPS, and if the blackout lasts long enough to drain the UPS battery, monitoring software can effect an orderly shutdown of the node(s) to minimize possible file system damage and facilitate a clean reboot process later.
You may purchase UPS systems from Aspen that are integrated into the cluster rack(s) and monitored by ABC. If your nodes are not protected by UPS, then current jobs running when the power outage occurs will fail. It is possible to equip every node in your cluster with UPS protection so that jobs are not interrupted by power outages, however this will adversely affect your rack space utilization (UPS systems can be up to 5U in height) and can be costly, depending on the number of nodes in your cluster. Unless ABC is used, some complexity is introduced into your system in monitoring and controlling multiple UPS systems with the same master (an SNMP management card is required in each UPS in most cases), and we will need to carefully balance your run times for each UPS under normal load conditions by moving nodes from one UPS to another.
Additional UPS runtime can be configured for your cluster by adding additional battery packs to your UPS system(s) as well. Speak to your Aspen sales engineer about your specific UPS needs.
In North America, your facility may provide A.C. power at 120 volts or 208 volts. There are reasons to prefer one voltage over another.
A.C. Power - 120v Circuits
Convenience and circuit availability are the most usual reasons to power your cluster with 120v circuits. 120 volt (v) 15 amp (a) office power outlets are ubiquitous in almost all environments, and if you are installing your cluster in a converted room that does not have raised floor or was not designed specifically for computers, these circuits will most likely already exist in your facility. However, there are significant limitations to the amount of power available on these circuits, and you may require more of these circuits than you have available.
The majority of existing office 120v circuits are rated at 15a. The Underwriters Laboratory specifies that a circuit can not draw more than 80% of a receptacles rating for safety, leaving 12 amps available.
Total watts available is derived by multiplying the voltage (120) by the amperage (12), so we have the equation;
120(v) x 12(a) = 1440(w)
This means that a standard 15a receptacle can provide 1440 Watts. Now, two terms are used to describe electrical equipment power ratings. “Watts” are the true, or maximum, power of the circuit, and “volt-amps” is the apparent power, or what the circuit can really produce given the effects of capacitance and inductance by the components in the load. Watts are the maximum the circuit can produce, while Volt-amps are the actual power that the circuit can produce under real world circumstances, and depends on what type of equipment is attached to the circuit, and what that equipments “power factor” is.
Most Aspen servers have power factor corrected power supplies with a .85 or higher correlation between Volt Amps and Watts, however some inevitable internal power supply loss occurs. We'll use a power factor of .95 to allow for lower efficiency in some of the electrical devices in the cluster.
Total volt-amps is derived by multiplying the voltage (120) by the amperage (12) by the power factor (.95), so we have the equation;
120(v) x 12(a) x .95(pf) = 1368(va)
This 120v 15a circuit can provide approximately 1368 volt-amps of actual power. 120v 20a (1824 volt-amps) and 120v 30a (2736 volt-amps) rated circuits do exist in some facilities, although they are a bit less common.
Most 1U nodes that Aspen sells are equipped with a 650 watt power supply, This is the maximum power the node power supply can draw, but a nodes true average running load is between ~300 and ~322 watts (equipped with a single hard drive and 8 GB memory) under load. Additional memory or hardware options, or processor intensive code execution, can increase this load significantly. As a general rule of thumb, no more than 4 1U compute nodes should be connected to any single 120v 15a circuit (1368 / 322 = 4.2).
Your master or storage nodes, or “twin” systems (two systems in 1U sharing a power supply) have 800 to 1200 watt power supplies. A master or storage node, and its associated external RAID if so equipped, can draw up to ~600 watts under nominal load due to being equipped with additional memory and disks as well as RAID cards and other accessories. If your master or storage node has redundant power supplies, you should budget the total power load on every circuit that powers one of the power supplies.
Your Aspen sales engineer can provide much more exact numbers, and will do so when he or she computes the power load for your configured cluster. Aspen also provides a Facilities Estimator that will estimate your power needs based on your selections.
Please note that if you intend to over-subscribe your cluster or routinely perform processor intensive calculations, your power demands will be higher.
A.C. Power - 208v Circuits
Power capacity is one major reason to power your cluster with 208v circuits. Almost all node power supplies are capable of auto-switching between 120v and 208v power sources, and common 208v circuit sizes are 20a (3161va) and 30a (4742va) , but even larger circuits are available. Any node will draw less current (~½) at 208v than it will at 120v, reducing waste heat and increasing power supply component life, and allowing more nodes to be fed from a single circuit and its associated UPS or PDU.
It is also common for each 208v receptacle to have its own circuit breaker, whereas 120v 15a circuits are often wired with multiple outlets per circuit. Many people have probably had the experience of turning on one appliance in an older house which blew the circuit breaker, only to find that electrical devices in a totally different room became non-functional. This is not a good thing to have happen to a circuit that is powering part of your cluster.
208v circuits are superior to most 120v circuits in other ways as well. 208v circuits are commonly equipped with twist lock plugs, which are turned, or twisted, to lock into the receptacle. Although 120v twist lock plugs do exist, many 120v circuits employ straight plugs, which do not lock. If your plugs are not twist lock, an inadvertent bump or cable pull can remove power from parts of your cluster, especially in crowded underfloor conditions. 120v electrical service components are often “consumer” grade, while 208v receptacles are most likely industrial grade, so the quality of 208v electrical components are higher, reducing the chance of intermittent power connection issues.
240v Circuits, 3-phase Supply, or Direct Current (D.C.) Feed
In some unusual cases, your facility may provide 240v circuits to supply your compute power needs. Virtually all single phase 208v equipment will operate from a 240v circuit, and your cluster can almost always be configured identically to one powered from 208v circuits.
Your facility may provide 3-phase power (a power feed characterized by 3 “legs” which provide their peak power at different times in the cycle). In this case, the power must be converted to single phase (a 3-phase 208v 50a circuit will convert to 3 separate 120v 50a single-phase to neutral circuits or a combination of double pole 208v and 120v circuits) either by your facility infrastructure, or via a converter Aspen can configure for your cluster.
While much more common in telecommunications facilities, your facility might provide only D.C. power for for your computing needs. Normally, D.C. power is used in facilities with battery backup and/or a generator, as its use can simplify power generation and storage. Specialized node power supplies, along with inverters for peripherals and devices which cannot be configured with D.C. power supplies, are necessary to configure a fully D.C. powered cluster, and not all node configurations are supported. Aspen can help you with your D.C. powered computing needs.
International Power Standards
Many countries have different power options, plug configurations, and connection requirements. Aspen builds systems that are installed in countries all over the world, and can configure your cluster so that it matches the power supplied in your facility. Discuss any specific power connection requirements you may have for your cluster with your sales engineer.
What power options to use
Use 208v circuits:
- whenever possible and economically feasible
- if you have to install additional power circuits to provide power to your new cluster, and your facility and electrical panels will support the addition of 208v circuits
Use 120v circuits only if:
- a sufficient number of existing 120v circuits already exist in your facility to power your cluster, or
- you will have to install additional power circuits to provide power to your new cluster, and your facility can not provide a sufficient number of 208v circuits in your facility electrical panels to support your cluster
Use 240v circuits only if your facility is equipped only with 240v infrastructure. Work with your Aspen sales engineer to determine interoperability of all selected cluster components with the 240v power source. Use 3-phase to single phase conversion only if your facility provides only 3- phase power. The additional equipment needed for 3-phase to single phase conversion can be expensive. Use D.C. as your cluster power source only if that is the only power source available in your facility, as that option will limit your hardware configuration choices.
In all cases, your Aspen sales engineer will speak with you to understand your facility power options, and configure the cluster to that specification. They will also provide you with the number and type of circuits required. Aspen recommends using 208v single phase power, and 30 amp or larger circuits for your cluster(s) power requirements.
Plug Types
One of the most common customer errors is to specify incorrect plug types for their cluster installation. International customers do not have this problem as often, as they are accustomed to stating specifically what standard plug type they require.
National Electrical Manufacturers Association (NEMA) plugs and receptacles are commonly used in North America, and use designators such as “NEMA L6-30R” to identify receptacle and plug types. The “R” stands for receptacle, which is the receptacle you provide at your facility to plug your cluster into, while “P” stands for plug.
The most common plug/receptacle types you will encounter in North America are;
120v – 15a to 30a
- NEMA 5-15r (120v 15a standard office wall plug)
- NEMA 5-20r (120v 20a circuit installed in some more modern facilities)
- NEMA L5-20r (120v 20a twist lock receptacle)
- NEMA L5-30R (120v 30a twist lock receptacle)
208v – 20a to 50a
- NEMA L6-20R (208v 20a twist lock)
- NEMA L6-30R (208v 30a twist lock)
- NEMA L14-30R (208v 30a twist lock 4-Wire Grounding)
- CS6364 (208v 50a twist lock “California Style” 4-wire Grounding)
The CS6364 receptacle and plug combination is not a NEMA designation, but is becoming more common as power needs grow. There are many other types of electrical receptacles and plugs. For instance, the back of most nodes are equipped with International Electrotechnical Commission (IEC) C14 chassis plugs, while the power cords connecting the node to its PDU are normally C13 line socket to NEMA 5-15P.
You may need to consult with your electrical personnel at your facility to determine exactly what receptacles you have or can support. In all cases, speak with your sales engineer if you are confused about your options or what type of receptacles your current facility has.
