Monkey Talk: Cluster Opinions and Insights from Cluster Monkey.
By Douglas Eadline
So you want to build a cluster. First let's take a look at the good news. There are plenty of options in choosing both hardware and software. Now let's take a look at the bad news. There are plenty of options in choosing both hardware and software. And, as I have said in the past, when mistakes are made with clusters the problem is multiplied by the number of nodes.
The really bad news is one wrong decision may cause you to get less than the full potential of your cluster. In the past, I have seen cases where in order to save $20 a node, a lower cost network card was used. As it turned out, the network card was not very good and a little math showed that by lowing the price by less than one percent the designers were able to reduce the cluster performance by 30% percent. Whoops.
The lesson is quite simple. The design metrics needed for clusters do not come from glossy data sheets. Unfortunately, until you put a cluster together, you may not know how well everything will function as a whole. So now the question becomes How do I test something that does not exist?. There is no easy answer to this question. There are some things you can do, however. First, visit the what I call the Clustanator at the Agregate.org Your time will be well spent. Second, ask vendors for benchmark data. On ClusterMonkey there are articles about freely available tools that allow you test various aspects of your cluster.
I also wanted to mention the status of the ClusterWorld Benchmarking Project (CWBP). The goal of this project is to provide a set of benchmarking tools that will allow clusters to be developed and optimized for specific application areas and work flows. Unfortunately, ClusterWorld Magazine ceased publication in June of this year and the project has slowed. Now that Cluster Monkey is up and running, the project will be continuing as the ClusterMonkey Benchmarking Project (CMBP) -- flip the W over. I'll be talking more about this in the near future.
In the mean time, remember, faster, better, cheaper only really works when you test, design, and optimize your cluster and application. And, don't forget that once you know your cluster design, use the Request a Quote form at LinuxHPC.org
Douglas Eadline can be found swing around the binary trees at Cluster Monkey
This work licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 License