Monkey Talk: Cluster Opinions and Insights from Cluster Monkey.

By Douglas Eadline

The term disruptive technology is often used to describe a new technology that undermines the current status quo, puts companies out of business, and pretty much changes the way things are done. Recent at SC05 IDC reported that over the last five years the use of HPC clusters have exceed IDC's optimistic projections. In the last two years alone, clusters have grown from a one third market share to encompass almost half the market. Large capability systems (heroic supercomputers) have seen a decrease in market share. HPC clusters are disruptive.

What about when a disruptive technology is really disruptive? I do not mean it does any physical damage, but what about when it changes things so much that the market has trouble recovering. I think HPC clusters may be suffering from this effect because they are the result of three powerful disruptive forces; commodity hardware, open software, and the Internet -- an almost perfect disruptive storm if you will.

To describe the effect, I like to talk about the what I call the Beowulf Bomb. Prior to the Beowulf/Commodity approach, HPC systems were highly integrated big iron affairs. Expensive as they were, there was a least some effort to make it all work together and provide blanket support. The advent of the Beowulf approach dropped a bomb into these systems. Now each component is sourced from a different vendor and the core software is openly available. The cost dropped, users gained much more control, and more people could now play the game, but there was collateral damage. It would seem that these marvelous scalable systems were not very scalable in a marketing sense. The promised panacea of cheap ubiquitous supercomputing still relies on custom and often tedious integration and thin support. Of course, some vendors will argue that they have this solved with blades and clever packaging schemes, but the reality is that each cluster is often a unique solution designed to! solve hard problems using hardware and software from many different sources. This situation leads to what I consider one of the market growth inhibitors -- disrupted support.

There are some other choices, however. There are "integrated clusters" from larger vendors that reduce the number of user options in order to increase the level of performance, integration, and support. For some users this is a cost effective trade-off. For others, more integration means more engineering and therefore a higher cost.

There is little doubt that commodity based solutions with various levels of integration will continue to win the day even with the issues mentioned above. In closing, I believe that disruptive technology storms provide milestones in the market that invite us to clear the ruble from our desks, take out a clean sheet of paper, and re-think that which we thought was not possible -- blue skies and all.

Douglas Eadline can be found swing around the binary trees at Cluster Monkey

This work licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 License