How to think outside the box: The awful burden of legacy tech | ZDNet

In 1998 I was part of a Sun Microsystems team sent to Seattle to meet an online bookseller with a problem. They were close to maxing out the largest IBM mainframe and soon their business would come to a screeching halt. Could Sun help?

Sun had a powerful multi-processor RISC server, the E10000. And we had a fabric-based storage system awaiting the release of the first Fibre Channel switch to provide unheard of storage scalability. But it was nowhere near enough.

We – and the rest of the legacy computer industry – came up empty. We had no idea how to even approach solving their problem.

Amazon finally hired a professor to lead the creation of their own solution: an immensely scalable, shared-nothing cluster, with fully distributed computers and storage linked over an Ethernet network overlaid by a software virtualization layer that made it all look like a single server. That was the genesis of Amazon Web Services, Amazon’s profit engine today. 

Why couldn’t we see a solution?

In retrospect, all the architectural elements of a warehouse-scale computer – as Google would call it – were available. 

Software virtualization of hardware resources had been well-proven by the late 1970s. In the 1980s, DEC’s VAXclusters had been a popular solution for high availability and scalable enterprise computing. In 1997 Microsoft released their shared-nothing Wolfpack cluster. 

Web architectures used load-balancing front-ends with pizza-box servers on the back-end. VMware, founded early in 1998, had begun to make a business out of replacing thousands of physical servers with thousands of virtual servers. 

The legacy industry’s problem was two-fold. First, we simply hadn’t encountered a web-scale customer that needed huge amounts of easily expanded compute and storage. 

Second, and more importantly, we were used to thinking inside a box. Boxes with 60- to 70-percent gross margins and 20-percent annual maintenance. 

Not cheap commodity boxes with PC disk storage, 100Mbit Ethernet interconnects, and an integrated software layer tying them all together, hiding their failures while optimizing their performance. We not only couldn’t conceive of such an architecture, we would have to fire our costly direct sales forces to sell them. 

As Upton Sinclair said, “It is difficult to get a man to understand something, when his salary depends on his not understanding it.”

And exactly none of the legacy box companies have become leaders in cloud computing. 

What I learned about boxes

To think outside the box, you first have to find the box. Explore its limits. What assumptions undergird it. How it interacts with your business model and worldview to keep you from seeing beyond it. It’s a really hard problem.

That’s why it took academics to solve the problem. People who weren’t married to the existing box, and had the expertise to create the first proof of concept. Then many people could think inside a new box.

The Take

Some people innovate because they don’t know it can’t be done. And there are those – like Amazon and Google – who sit outside the box and reframe the problem, usually at the intersection of technology and economics.

Assuming you can think, getting outside the box is job #1. That’s why innovators seem to come out of left field, because everyone else is on the inside looking out. 

That’s how disk storage evolved from 24-inch disks to 2.5-inch disks, crushing multiple generations of once successful vendors with each new — and perfectly predictable — form factor. Winnowing more than 200 vendors to three inside steadily shrinking boxes.

Comments welcome. What are your experiences with boxes – either from the inside or the outside?