Source
When Sony issued a recent PlayStation 3 update removing the device's ability to install alternate operating systems like Linux, it did so to protect copyrighted content—but several research projects suffered collateral damage.
The Air Force is one example. The Air Force Research Laboratory in Rome, New York picked up 336 PS3 systems in 2009 and built itself a 53 teraFLOP processing cluster. Once completed as a proof of concept, Air Force researchers then scaled up by a factor of six and went in search of 2,200 more consoles (later scaled back to 1,700). The $663,000 contract was awarded on January 6, 2010, to a small company called Fixstars that could provide 1,700 160GB PS3 systems to the government.
Getting that many units was difficult enough that the government required bidders to get a letter from Sony certifying that the units were actually available.
Dirt cheap computing
Another grotesque waste of taxpayer dollars? Exactly the opposite, according to research lab staff. Off-the-shelf PS3s could take advantage of Sony's hardware subsidy to get powerful Cell processors more cheaply than via any other solution.
"The Advanced Computing Architectures team at the Information Directorate considered several alternatives to arrive at the configuration of the proposed system, including the Sony BCU-100, IBM Blade Q22, and IBM PowerXCell 8i CAB accelerators cards," said the Air Force last year. "In particular, the performance capabilities of the Cell Broadband engine were examined in considerable detail on each of the algorithms."
The team also looked into using dual-quad-core Xeon servers for its cluster, going so far as to do a "detailed study of Xeon multithreading and SSE4 optimization on image processing intensive tasks." The hardware worked well, and it eventually came to serve as subcluster headnodes that sit between the PS3 cluster itself and the control terminals.
But building the entire cluster out of Xeons would cost "more than an order of magnitude greater than the PS3 technology." The team also looked into advanced GPGPUs but found that they worked best to "accelerate a subset of our algorithms, particularly the frontend processing and backend visualization, but lag the PS3 in the bulk of the calculations where processes need to intercommunicate and share memory beyond what is supported efficiently by the GPGPUs."
The initial test cluster (source: US Air Force)
The result was the 500 TeraFLOPS Heterogeneous Cluster powered by PS3s but connected to subcluster heads of dual-quad Xeons with multiple GPGPUs.
The Air Force team ordered the hardware, spent days unboxing it and imaging each unit to run Linux, and then... Sony removed the Linux install option a couple months later. (One can only imagine what happened to those 2,000 PS3 controllers and other unneeded accessories.)
Does it matter?
Sony's decision had no immediate impact on the cluster; for obvious reasons, the PS3s are not hooked into the PlayStation Network and don't need Sony's firmware updates. But what happens when a PS3 dies or needs repair? Tough luck.
We checked in with the Air Force Research Laboratory, which noted its disappointment with the Sony decision. "We will have to continue to use the systems we already have in hand," the lab told Ars, but "this will make it difficult to replace systems that break or fail. The refurbished PS3s also have the problem that when they come back from Sony, they have the firmware (gameOS) and it will not allow Other OS, which seems wrong. We are aware of class-action lawsuits against Sony for taking away this option on systems that use to have it."
A similar issue will confront academic PS3 clusters, which have sprung up in labs across the country. In 2007, a North Carolina State professor built himself a small cluster that he cobbled together after "he spent a few hours one day in early January driving from store to store to purchase the eight machines."
The University of Massachusetts has 16 machines networked into a cluster called the "Gravity Grid," used to look at gravitational waves and black holes. According to the physicists at UMass, the PS3's "incredibly low cost make it very attractive as a scientific computing node, i.e., part of a compute cluster. In fact, it's highly plausible that the raw computing power-per-dollar that the PS3 offers is significantly higher than anything else on the market today."
All such projects will last as long as the machines survive or used machines are still available, but new hardware can't be added and refurbished machines can't be used. A class-action lawsuit has recently targeted Sony for removing a promised feature retroactively, though the issue is unlikely to be decided anytime soon.
We asked Sony for comment on how its decision would affect scientific computing clusters, but received no answer before publication.
A love affair with off-the-shelf consumer hardware
Such are the dangers of relying on consumer-grade hardware sold with a very different set of concerns from those that bedevil the scientists, especially in an era where firmware updates routinely alter functionality. But the Air Force, for one, has no plans to stop.
"The gaming and graphics market continues to push the state of the art and lowers the cost of High Performance Computing, FLOPS/WATTS per dollar," the Air Force Research Laboratory told Ars. "This is important for embedded HPC, our area of expertise.
"The HPC environment is rapidly changing; leveraging technology that is subsidized by large consumer markets will always have large cost advantages. This gives us the experience (lesson learned) to develop HPC with low-cost hardware, benefitting the tax payer, Air Force, Air Force Research Lab while utilizing limited DoD budgets."